Installing Ollama

This guide explores the local LLM ecosystem and Ollama's place within it. The AI landscape includes cloud-based services like ChatGPT and local solutions that offer privacy, cost savings, and control. Local LLM tools function through inference engines (Ollama, LM Studio), various model formats (GGUF, GGML), and different user interfaces. Ollama stands out as an open-source tool that simplifies running large language models locally on personal computers. It provides a user-friendly interface for model management, enabling tasks like text generation, summarization, and code completion without cloud dependencies. While LM Studio offers a full GUI experience and LocalAI focuses on API compatibility, Ollama balances simplicity with power through efficient CLI and basic web interfaces..

AI Ecosystems and Local LLM Tools

The AI ecosystem for large language models (LLMs) consists of two primary deployment approaches: cloud-based and local. Cloud-based solutions like OpenAI’s ChatGPT, Claude, and Google’s Gemini offer powerful capabilities but come with subscription costs and data privacy considerations. Local LLM tools have emerged as alternatives that provide greater control over data, reduced costs, and customization options.

Within the local LLM ecosystem, several tools enable users to run AI models on their personal computers:

1. Inference Engines: Software like Ollama, LM Studio, and LocalAI that handle the actual execution of models
2. Model Formats: Different standards like GGUF, GGML, and PyTorch formats that define how models are stored and loaded
3. User Interfaces: Various ways to interact with models through CLI, GUI, web interfaces, or API endpoints

Ollama fits into this ecosystem as a leading inference engine that simplifies model management and provides an API for integrations.

Popular Local LLM Tools

LM Studio

LM Studio is a desktop application designed to provide an intuitive graphical interface for running LLMs locally. Key features include:

• GUI-based model management and inference
• Support for GGUF format models
• Built-in model browser for downloading models from Hugging Face
• Chat interface with conversation history
• OpenAI-compatible API for integration with other applications
• Advanced inference parameter controls
• Support for Windows, macOS, and Linux

LocalAI

LocalAI is an open-source, self-hosted alternative to the OpenAI API that supports various models and architectures:

• OpenAI API compatibility for drop-in replacement
• Support for multiple model formats (GGUF, GGML, PyTorch)
• Multi-modal capabilities (text, image, audio)
• Container-friendly design for easy deployment
• Function calling and tools API

Text Generation WebUI

A comprehensive web interface for running LLMs with extensive features:

• Web-based UI accessible from multiple devices
• Support for many model architectures and formats
• Extensions ecosystem
• Character and persona creation tools
• Training and fine-tuning capabilities

Koboldcpp

A lightweight C++ implementation focused on creative writing and storytelling:

• Optimized for narrative and creative text generation
• Low resource requirements
• Integrations with role-playing interfaces

Comparing Local LLM Tools

Similarities

Differences

Model Compatibility and Sharing

Model Formats

Different tools use different model formats:

1. GGUF (GPT-Generated Unified Format): Successor to GGML, used by Ollama and LM Studio, optimized for efficient inference on consumer hardware.

2. GGML (GPT-Generated Model Language): Older format still used by some tools, being phased out in favor of GGUF.

3. PyTorch/Safetensors: Native formats used by many AI research labs, less optimized for consumer hardware.

4. ONNX: Open standard for machine learning interoperability, supported by various tools.

Model Storage Locations

Model storage varies by tool:

• Ollama: Stores models in ~/.ollama/models on Linux/macOS and C:\Users\<username>\.ollama\models on Windows.
• LM Studio: Typically stores models in a user-configurable location, defaulting to ~/lmstudio/models on macOS/Linux.
• LocalAI: Stores models in its configured models directory, customizable at setup.
• Text Generation WebUI: Stores models in the models subdirectory of its installation.

Model Sharing Between Tools

Models can be shared between different tools with some limitations:

• GGUF models: Can generally be used across Ollama, LM Studio, and LocalAI, though parameter settings may need adjustment.
• Ollama specific models: Models pulled via Ollama may need to be extracted or converted before use in other tools.
• Custom formats: Some tools have proprietary enhancements or metadata that don’t transfer to other platforms.

To use the same models across tools:

1. Store models in a central location
2. Configure each tool to access this location
3. Ensure format compatibility (most tools now support GGUF)
4. Be aware that quantization levels and parameters may vary between tools

Understanding Hugging Face and Model Hubs

Hugging Face serves as the central hub for machine learning models - essentially the “GitHub of machine learning models.” It provides a collaborative platform where researchers and developers can share, discover, and use pre-trained models.

Key characteristics of Hugging Face include:

• Vast model repository: Hosts thousands of models for various AI tasks
• Multiple access methods: Models can be:
  • Downloaded manually through the website
  • Accessed via APIs using libraries like Transformers
  • Used directly by tools like LM Studio, KoboldCpp, and others
• Community contributions: Allows users to upload their own fine-tuned models
• Standardized formats: Primarily distributes models in formats like GGUF/GGML for efficient local inference

LM Studio primarily pulls models from Hugging Face in .gguf format, making it a cornerstone of the local LLM ecosystem’s model distribution infrastructure.

The Core Issue: Model Silos

A fundamental challenge in the local LLM ecosystem is that tools like Ollama and LM Studio use separate download systems and storage directories for LLMs. They do not share models by default, even if the same model has already been downloaded to your computer.

This creates “model silos” where:

1. Redundant storage: The same model might be stored twice in different locations
2. Format incompatibilities: Models downloaded for one tool often can’t be directly used by another
3. Inconsistent experiences: The same model might behave differently across tools due to different backends

Technical Reasons for Model Discrepancies

The technical reasons for these model discrepancies include:

1. Different formats and backends:

   • Ollama uses a custom model packaging format for optimized serving (typically .modelfile or .bin formats)
   • LM Studio and many other tools use GGUF or GGML formats (developed for the llama.cpp inference engine)

2. Isolated storage systems:

   • Tools don’t look into each other’s directories for model files by default
   • Each maintains its own metadata about models, making cross-tool discovery difficult

3. Runtime differences:

   • Ollama: Optimized C++ backend with custom format and API emphasis
   • LM Studio: llama.cpp-based with GGUF format and GUI focus

Advanced Solutions for Model Sharing

Best Practices for Model Interoperability

To maximize efficiency and avoid duplicating large model files, consider these approaches:

1. Choose a primary tool for model management:

   • Use LM Studio if you prefer a GUI, GGUF models, and local experimentation
   • Use Ollama if you want fast server-like local inference and better integration with CLI and APIs

2. Use Ollama’s API server approach:

   • Start Ollama with your preferred model: ollama run mistral
   • Connect other applications to Ollama’s API at http://localhost:11434
   • This lets you use one model instance across multiple interfaces

3. Use advanced configuration:

   • Some tools allow specifying alternative model directories
   • This can reduce duplication but requires technical configuration

Advanced Option: Converting Between Formats

For advanced users, it is theoretically possible (though complex) to convert between model formats:

1. GGUF to Ollama format:
   • Extract the GGUF model
   • Create a Modelfile defining the model’s parameters
   • Repackage using ollama create

However, this approach is not officially supported and may not work reliably due to backend differences and frequent updates to both tools and formats.

Ollama

Ollama is an open-source tool for running large language models locally. This guide covers setting up Ollama, pulling models from GitHub, and customizing models. It also explores REST APIs, Python, and JavaScript integration.

Key features of Olama

• Model Management: Easily download and switch between different large language models.
• Unified Interface: Interact with various models using a consistent set of commands.
• Extensibility: Support for adding custom models and extensions.
• Performance Optimizations: Utilize your own hardware effectively, including GPU acceleration if available.

Advantages of using Olama

• Enhanced privacy and security by running models locally.
• Simplified setup process for large language models.
• Cost efficiency by eliminating the need for cloud-based services.
• Reduced latency due to local execution.
• Greater flexibility in customizing or fine-tuning models.

Use cases for Olama

• Development and testing: Test different large language models to see which one performs better.
• Education and research: Easy platform for learning and experimentation.
• Secure applications: Suitable for industries like healthcare or finance where data privacy is critical.

System requirements for Olama

• Supports Mac, Linux, and Windows.
• At least 10 GB of free storage.
• Modern CPU processor.
• Optional: GPU for acceleration.

Olama provides a straightforward way to download, run, and interact with various large language models locally, making advanced language processing accessible to developers, researchers, and hobbyists.

Installation and Setup

1. Download Olama from the official website.
2. Install the command line interface.
3. Run the first model (e.g., Lama 3.2).
4. Explore different models and their features.

Quick Installation

• Run the following command to install Ollama:

 1# Run this command
 2~ curl -fsSL https://ollama.com/install.sh | sh                                                                             Fri 31 Jan 2025 18:08:54 GMT
 3>>> Installing ollama to /usr/local
 4[sudo] password for ghafoor:
 5>>> Downloading Linux amd64 bundle
 6######################################################################## 100.0%
 7>>> Creating ollama user...
 8[sudo] password for ghafoor:
 9>>> Adding ollama user to render group...
10>>> Adding ollama user to video group...
11>>> Adding current user to ollama group...
12>>> Creating ollama systemd service...
13>>> Enabling and starting ollama service...
14Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.
15>>> The Ollama API is now available at 127.0.0.1:11434.
16>>> Install complete. Run "ollama" from the command line.
17WARNING: No NVIDIA/AMD GPU detected. Ollama will run in CPU-only mode.

The above command downloads, installs, creates a new user ollama, adds the user to the render and video groups, and creates a systemd service for the ollama user. It also enables and starts the ollama service. The Ollama API is now available at port 127.0.0.1:11434. The command also warns that no NVIDIA/AMD GPU is detected, so Ollama will run in CPU-only mode.

Installing Ollam Manually

Manual-Installation of Ollama is also possible. The following steps are required to install Ollama manually:

1curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz
2sudo tar -C /usr -xzf ollama-linux-amd64.tgz

Start Ollama

• Run ollama serve
• It should start the server, as shown below:

 1Couldn't find '/home/ghafoor/.ollama/id_ed25519'. Generating new private key.
 2Your new public key is:
 3
 4ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINEfzjUOUtAIvUYXkUDA9RCxJwJOMQ8Pec2Ntwbjkxhq
 5
 62025/02/07 02:12:05 routes.go:1187: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/ghafoor/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
 7time=2025-02-07T02:12:05.401Z level=INFO source=images.go:432 msg="total blobs: 0"
 8time=2025-02-07T02:12:05.401Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
 9[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
10
11[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
12- using env: export GIN_MODE=release
13- using code: gin.SetMode(gin.ReleaseMode)
14
15[GIN-debug] POST   /api/pull                 --> github.com/ollama/ollama/server.(*Server).PullHandler-fm (5 handlers)
16[GIN-debug] POST   /api/generate             --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
17[GIN-debug] POST   /api/chat                 --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
18[GIN-debug] POST   /api/embed                --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (5 handlers)
19[GIN-debug] POST   /api/embeddings           --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
20[GIN-debug] POST   /api/create               --> github.com/ollama/ollama/server.(*Server).CreateHandler-fm (5 handlers)
21[GIN-debug] POST   /api/push                 --> github.com/ollama/ollama/server.(*Server).PushHandler-fm (5 handlers)
22[GIN-debug] POST   /api/copy                 --> github.com/ollama/ollama/server.(*Server).CopyHandler-fm (5 handlers)
23[GIN-debug] DELETE /api/delete               --> github.com/ollama/ollama/server.(*Server).DeleteHandler-fm (5 handlers)
24[GIN-debug] POST   /api/show                 --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (5 handlers)
25[GIN-debug] POST   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
26[GIN-debug] HEAD   /api/blobs/:digest        --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
27[GIN-debug] GET    /api/ps                   --> github.com/ollama/ollama/server.(*Server).PsHandler-fm (5 handlers)
28[GIN-debug] POST   /v1/chat/completions      --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
29[GIN-debug] POST   /v1/completions           --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (6 handlers)
30[GIN-debug] POST   /v1/embeddings            --> github.com/ollama/ollama/server.(*Server).EmbedHandler-fm (6 handlers)
31[GIN-debug] GET    /v1/models                --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (6 handlers)
32[GIN-debug] GET    /v1/models/:model         --> github.com/ollama/ollama/server.(*Server).ShowHandler-fm (6 handlers)
33[GIN-debug] GET    /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
34[GIN-debug] GET    /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
35[GIN-debug] GET    /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
36[GIN-debug] HEAD   /                         --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
37[GIN-debug] HEAD   /api/tags                 --> github.com/ollama/ollama/server.(*Server).ListHandler-fm (5 handlers)
38[GIN-debug] HEAD   /api/version              --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
39time=2025-02-07T02:12:05.403Z level=INFO source=routes.go:1238 msg="Listening on 127.0.0.1:11434 (version 0.5.7)"
40time=2025-02-07T02:12:05.414Z level=INFO source=routes.go:1267 msg="Dynamic LLM libraries" runners=[cpu]
41time=2025-02-07T02:12:05.414Z level=INFO source=gpu.go:226 msg="looking for compatible GPUs"
42time=2025-02-07T02:12:05.442Z level=INFO source=gpu.go:392 msg="no compatible GPUs were discovered"
43time=2025-02-07T02:12:05.442Z level=INFO source=types.go:131 msg="inference compute" id=0 library=cpu variant="no vector extensions" compute="" driver=0.0 name="" total="47.0 GiB" available="38.8 GiB"
44[GIN] 2025/02/07 - 02:13:18 | 200 |    2.686714ms |       127.0.0.1 | GET      "/api/version"

Interact with Ollama server

Open another terminal and and run ollma -v to find out the version of Ollama. The output should be similar to the following:

 1> ollama -v
 2ollama version is 0.5.7
 3# create user name and group for ollama
 4>  sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama
 5>  sudo usermod -a -G ollama $(whoami)
 6[sudo] password for ghafoor:
 7# Create a service file in /etc/systemd/system/ollama.service
 8> sudo nano /etc/systemd/system/ollama.
 9# Add the following content to the file
10[Unit]
11Description=Ollama Service
12After=network-online.target
13
14[Service]
15ExecStart=/usr/bin/ollama serve
16User=ollama
17Group=ollama
18Restart=always
19RestartSec=3
20Environment="PATH=$PATH"
21
22[Install]
23WantedBy=default.target
24# Enable the service
25> sudo systemctl
26> sudo systemctl enable ollama
27Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service.
28# Start the service
29> sudo systemctl demon-reload
30> sudo systemctl start ollama

Ollma Key Commands

• help: Displays available commands.
• show info: Provides information about the current model.
• list: Lists all installed models.
• remove: Removes a specified model.
• pull: Downloads a specified model.

Running a model with Ollama

• First get the commands of ollama to see what commands are available.
• Then run the model Llama3.2 using the command ollama run llama3.2.

 1ollama -h
 2
 3Large language model runner
 4
 5Usage:
 6  ollama [flags]
 7  ollama [command]
 8
 9Available Commands:
10  serve       Start ollama
11  create      Create a model from a Modelfile
12  show        Show information for a model
13  run         Run a model
14  stop        Stop a running model
15  pull        Pull a model from a registry
16  push        Push a model to a registry
17  list        List models
18  ps          List running models
19  cp          Copy a model
20  rm          Remove a model
21  help        Help about any command
22
23Flags:
24  -h, --help      help for ollama
25  -v, --version   Show version information
26
27Use "ollama [command] --help" for more information about a command.

To run a model with Ollama, use the following command:

 1ollama run llama3.2
 2pulling manifest
 3pulling dde5aa3fc5ff... 100%  2.0 GB
 4pulling 966de95ca8a6.... 1.4 KB
 5pulling fcc5a6bec9da... 100% 7.7 KB
 6pulling a70ff7e570d9... 100%  6.0 KB
 7pulling 56bb8bd477a5... 100% ▕96 B
 8pulling 34bb5ab01051... 100%  561 B
 9verifying sha256 digest
10writing manifest
11success
12>>> Send a message (/? for help)

Ollama Prompt

When you get this symbol >>>, it is called a prompt and you can communicate with the model using this prompt. You can type a message and press Enter to get a response from the model. For example, you can ask the model to generate text, answer questions, or provide summaries.

 1Are you there ?
 2Yes, I'm here. How can I assist you today?
 3
 4>>> I am fine, just wondering if you know anything about English Grammar?
 5English grammar is a vast and complex topic, but I'd be happy to help with any questions or topics you'd like to explore.
 6
 7What specific areas of English grammar are you interested in learning more about? Here are some popular areas:
 8
 91. Verb tenses: Present, past, and future tenses.
102. Clauses and phrases: Independent and dependent clauses, phrase structures, etc.
113. Sentence structure: Simple, compound, complex sentences.
124. Modality: Modal verbs (e.g., can, could, may, might), modal adverbs, etc.
135. Pronouns: Types of pronouns, their usage, and functions.
146. Nouns: Parts of speech, noun types (common, proper, collective, etc.), and grammar rules.
157. Adjectives: Modifiers, adjective placement, and usage.
16
17Or perhaps you have a specific question or issue with English grammar?
18
19Feel free to ask, and I'll do my best to help!
20>>>

Issues Faced

1. Very slow responses from the model.
2. When only this question was asked >>> Are you there?, the model responded with Yes, I'm here. How can I assist you today? . It generated the output very slowly and in doing so all 8 cpus were fully utilised but not the memory. The model was not able to generate the output quickly.

How to Fix the Issue

1. Use a more powerful machine equipped with modern CPUs and graphic Cards to use GPUs for faster processing.
2. Configure more threads to be used by the tool which is using a model.
3. Watch for memory usage, if needs more increase it.
4. Batch processing can be used to process multiple requests at once, if it allows resulting in parallel processing.
5. Use a different model or architecture that is better suited for your specific use case.

Technical Considerations

Optimize Context Window size, Embedding size, Quantization and Temperature to get better performance.6. Use REST APIs to interact with the model and get responses.

• Context Window Size The context window size determines how much of the conversation history the model considers. A larger window improves contextual awareness but increases resource usage.

• Embedding Size The embedding size is the vector representation of each token. A larger size captures more nuances but requires more resources.

• Quantization Quantization reduces model precision to improve speed and efficiency, though it may slightly reduce accuracy.

• Temperature Temperature controls the randomness of responses. Lower values produce more conservative outputs, while higher values yield more creative results.

Conclusion

Ollama is a powerful tool for running large language models locally, offering flexibility and control. By understanding parameters like context window size, embedding size, quantization, and temperature, you can optimize model performance for your needs. Whether for development, research, or secure applications, Ollama provides a robust platform for experimenting with LLMs.