How to run a local LLM with Ollama and call it from your code

To run a local LLM with Ollama, install it with one command, then type ollama run gemma4 to download a model and chat with it. Ollama also starts an HTTP server on localhost:11434, so the same model answers curl, Python, JavaScript and any OpenAI client you point at it. The parts most guides skip are below: the API, model sizing, disk locations and the errors you are likely to see.

How do you install Ollama?

The Ollama README (opens in a new tab) lists a one-line installer for each desktop platform. Ollama is MIT licensed.

macOS and Linux

curl -fsSL https://ollama.com/install.sh | sh

On macOS you can also download Ollama.dmg and drag the app into Applications. The macOS docs require Sonoma (version 14) or newer. Apple M-series Macs get CPU and GPU support; Intel Macs run on the CPU only.

Linux users can install manually from a tarball and run Ollama as a systemd service. The Linux guide covers separate packages for AMD ROCm and ARM64.

Windows

irm https://ollama.com/install.ps1 | iex

The alternative is OllamaSetup.exe. The Windows docs list Windows 10 22H2 or newer, NVIDIA driver 551.61 or newer for NVIDIA cards, and a ROCm v7 or Vulkan-capable driver for AMD Radeon. The installer does not need Administrator rights and puts Ollama in your home directory. Plan for at least 4 GB for the binaries, before any models.

Once installed, Ollama runs in the background on macOS and Windows. Check that the CLI works:

ollama -v

Pull and run your first model

ollama run downloads the model if it is missing, loads it and opens an interactive prompt:

ollama run gemma4

To download without chatting, use ollama pull. A name without a tag resolves to latest, so gemma4 and gemma4:latest are the same model. Pick a specific size by adding its tag, such as qwen3:8b.

ollama pull qwen3:8b
ollama ls
ollama ps

ollama ls lists what is on disk. ollama ps lists what is loaded in memory, how much memory each model takes and whether it sits on the GPU or CPU. Inside the chat prompt, wrap multi-line input in triple quotes (""").

Ollama keeps a model in memory for 5 minutes after the last request. ollama stop gemma4 unloads it immediately.

How much memory do you need to run a local LLM with Ollama?

Ollama's documentation does not publish a RAM table per model. What it does give you is the download size of every tag in its model library, and a tool (ollama ps) that shows where a loaded model ended up.

The tags for one model family show how wide the range is. Sizes below are as listed in the Ollama library for qwen3 on 4 October 2026, and are illustrative of how tags differ:

The download size is only the starting point. The context window needs memory on top of it, and the Ollama docs say a larger context length increases the memory a model requires.

Sizing checklist

Copy this and work through it before you pull a large model:

• ☐To do: Note your free VRAM (GPU) or free system RAM (CPU only, or Apple silicon).
• ☐To do: Pick a tag whose download size is clearly below that number, leaving room for context.
• ☐To do: Check the default context: Ollama uses 4k tokens below 24 GiB of VRAM, 32k from 24 to 48 GiB, and 256k at 48 GiB or more.
• ☐To do: If you raise the context (for agents and coding tools the docs suggest at least 64,000 tokens), budget more memory.
• ☐To do: If you raise OLLAMA_NUM_PARALLEL, budget more again: required RAM scales with parallel requests times context length.
• ☐To do: Load the model and run ollama ps. 100% GPU means it fits. A split such as 48%/52% CPU/GPU means part of it spilled into system RAM and will run slower.
• ☐To do: If it does not fit, drop to a smaller tag or try a quantized K/V cache (OLLAMA_KV_CACHE_TYPE=q8_0 roughly halves cache memory compared with the default f16).

For supported cards, the Ollama hardware support page (opens in a new tab) lists NVIDIA GPUs from compute capability 5.0 with driver 550 or newer, plus AMD cards through ROCm and Vulkan.

Calling the Ollama REST API with curl

Ollama's API listens on http://localhost:11434. The two endpoints you will use most are /api/generate for a single prompt and /api/chat for a message list. Both stream JSON objects by default; set "stream": false to get one response object.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{ "role": "user", "content": "Why is the sky blue?" }],
  "stream": false
}'

Model parameters go in an options object. This request sets the context window for one call:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Summarize the GPL in two sentences.",
  "stream": false,
  "options": { "num_ctx": 8192 }
}'

Other endpoints in the Ollama API reference (opens in a new tab) cover model management: GET /api/tags (local models), GET /api/ps (loaded models), POST /api/pull, DELETE /api/delete and POST /api/embed for embeddings. The keep_alive field controls how long the model stays loaded after a request: 0 unloads it straight away and -1 keeps it loaded.

For native clients, Ollama publishes pip install ollama for Python and npm i ollama for JavaScript.

Using the Ollama OpenAI-compatible endpoint

Ollama also serves a subset of the OpenAI API under /v1, so code written for the OpenAI SDK can run against a local model by changing two values. The client still requires an API key string, which Ollama ignores.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the client, ignored by Ollama
)

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(resp.choices[0].message.content)

According to the OpenAI compatibility docs (opens in a new tab), the supported routes are /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings and /v1/responses. Chat completions support streaming, JSON mode, vision and tools; logprobs are not supported. The Responses API arrived in v0.13.3 and is stateless only, so previous_response_id does not work.

Two details trip people up. You must ollama pull the model first, because the OpenAI routes do not download anything. The OpenAI API also has no context-size field, so for a bigger window you create a variant with a Modelfile:

FROM qwen3:8b
PARAMETER num_ctx 32768

ollama create qwen3-32k -f Modelfile

Then call qwen3-32k as the model name. If a tool insists on an OpenAI model name, ollama cp qwen3:8b gpt-3.5-turbo makes a copy under that name.

Where Ollama stores models and how to remove them

Models live in these directories by default, per the Ollama FAQ (opens in a new tab):

Set the OLLAMA_MODELS environment variable to move them, for example to a larger drive. On Linux with the standard installer, the ollama user needs read and write access: sudo chown -R ollama:ollama <directory>. On Windows, quit the tray app after setting the variable and start Ollama again.

Remove a model with the CLI or the API:

ollama rm qwen3:8b
curl -X DELETE http://localhost:11434/api/delete -d '{"model": "qwen3:8b"}'

The Windows uninstaller does not delete models stored in a custom OLLAMA_MODELS location, so clear that folder yourself if you want the space back.

Common Ollama errors and what they mean

Errors that happen mid-stream arrive as a final JSON line with an error property, and the HTTP status stays 200 because the response had already started. Check for that field in streaming code.

Logs are the next step. On macOS run cat ~/.ollama/logs/server.log. On Linux with systemd, use journalctl -u ollama --no-pager --follow --pager-end. Windows keeps server.log in %LOCALAPPDATA%\Ollama. Setting OLLAMA_DEBUG=1 adds detail, as described in the Ollama troubleshooting guide (opens in a new tab).