Serve an open model with vLLM behind an OpenAI-compatible API

To run a vLLM OpenAI compatible server, install vLLM on a Linux machine with a supported GPU, then start it with vllm serve <hugging-face-model-id>. The server listens on port 8000 and implements /v1/chat/completions, /v1/completions, /v1/models and more. Existing OpenAI clients work after you change the base URL and API key.

What vLLM does and when it is worth it

vLLM (opens in a new tab) is an Apache-2.0 library for LLM inference and serving. It started in the Sky Computing Lab at UC Berkeley and is now maintained by a community the README puts at over 2,000 contributors. The latest release at the time of writing is v0.30.0 (22 September 2026).

The project is built for serving traffic on accelerators. Its README lists PagedAttention for managing attention key and value memory, continuous batching of incoming requests, prefix caching, speculative decoding, and tensor, pipeline, data and expert parallelism across GPUs. It loads models straight from Hugging Face and supports over 200 model architectures, plus quantized formats such as FP8, INT4, GPTQ, AWQ and GGUF.

vLLM fits when several users or services share one GPU server and you want an OpenAI-shaped API in front of it. It is a heavier install than a desktop runner: Linux, Python and a GPU stack. For a single developer on a laptop, a local runner such as Ollama or llama.cpp needs less setup.

Hardware requirements from the vLLM docs

The GPU installation guide (opens in a new tab) lists these requirements:

• Operating system: Linux. vLLM does not support Windows natively; the docs point to WSL or community forks.
• Python: 3.10 to 3.13.
• NVIDIA: compute capability 7.5 or higher (T4, RTX 20xx, A100, L4, H100, B200 and newer). Pre-built binaries use CUDA 12.9, with CUDA 12.8 and 13.0 builds also published. Blackwell cards (B200, GB200) need at least CUDA 12.8.
• AMD: ROCm wheels support Python 3.12, ROCm 7.0 and glibc 2.35 or newer.
• Intel GPUs: supported through the XPU backend, with official Docker images since v0.26.0.
• Google TPU: a separate vllm-tpu package.
• Apple silicon: through the separate vLLM-Metal project, which uses MLX and needs MLX-format models.

vLLM also has CPU backends for x86, ARM AArch64, Apple silicon and IBM Z. Plugins outside the main repository cover Ascend NPUs and other accelerators.

How much VRAM you need depends on the model's weights, its context length and how many requests you batch. The docs give no per-model table, so size from the model card and use the memory flags below.

How do you install vLLM?

The quickstart recommends uv and a fresh environment, because vLLM's compiled CUDA kernels are tied to specific CUDA and PyTorch builds:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

--torch-backend=auto lets uv pick the PyTorch index that matches your installed CUDA driver. Plain pip works too:

pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

For AMD GPUs, install from vLLM's ROCm wheel index:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Running vLLM in Docker

The official image is vllm/vllm-openai on Docker Hub. Mounting the Hugging Face cache keeps downloaded weights between runs:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-0.6B

Any extra engine arguments go after the image tag. The --ipc=host flag (or --shm-size) gives PyTorch the shared memory it uses between processes, which matters most for tensor parallel inference.

Starting the vLLM OpenAI compatible server with vllm serve

The quickstart example starts a small instruct model:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

vLLM downloads the weights from Hugging Face on first run (set VLLM_USE_MODELSCOPE=True to use ModelScope instead). The quickstart (opens in a new tab) gives http://localhost:8000 as the default address and notes that the server hosts one model at a time.

Check it is up:

curl http://localhost:8000/v1/models

Two defaults surprise people. The server applies the model repository's generation_config.json if one exists, so sampling defaults may come from the model author rather than vLLM; pass --generation-config vllm to turn that off. The server also uses the chat template stored in the tokenizer unless you override it.

The OpenAI-compatible server docs (opens in a new tab) list the supported routes: /v1/completions, /v1/chat/completions, /v1/chat/completions/batch, /v1/responses, /v1/embeddings (embedding models), and /v1/audio/transcriptions and /v1/audio/translations (speech models). The README also mentions an Anthropic Messages API route.

Flags for GPU memory and context length

These are the flags that decide whether a model fits and how much traffic it can take. Defaults come from the vLLM engine arguments reference (opens in a new tab).

A few rules follow from the docs. --gpu-memory-utilization is per instance, so two vLLM servers sharing one GPU should each get about 0.5. Lowering --max-model-len and --max-num-seqs is one of the fixes the conserving memory guide suggests for out-of-memory errors, alongside quantized models, tensor parallelism and --enforce-eager. Setting --max-model-len auto makes vLLM use the model's full context if it fits, or the largest length that does.

A launch with explicit limits looks like this (illustrative values; size them to your model and GPU):

vllm serve Qwen/Qwen2.5-7B-Instruct \
    --max-model-len 16k \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 16 \
    --served-model-name qwen-7b

For a model too big for one GPU, add --tensor-parallel-size 4 to split it across four.

Calling vLLM with the OpenAI client

Point the official OpenAI SDK at the server. The model value is the Hugging Face ID, or whatever you passed to --served-model-name.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",  # must match --api-key if you set one
)

resp = client.chat.completions.create(
    model="qwen-7b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

Parameters that OpenAI does not support, such as top_k, go in extra_body:

resp = client.chat.completions.create(
    model="qwen-7b",
    messages=[{"role": "user", "content": "Name three rivers."}],
    extra_body={"top_k": 50},
)

Some request fields behave differently from OpenAI's API. The docs note that suffix is not supported on completions and user is ignored on chat completions. Tool calling needs the server started with --enable-auto-tool-choice and a --tool-call-parser that matches the model.

Securing the vLLM endpoint with an API key

Start the server with --api-key (or the VLLM_API_KEY environment variable). Clients then send the key as a Bearer token. You can pass several keys after the flag, which helps with key rotation.

vllm serve Qwen/Qwen2.5-7B-Instruct --api-key "$VLLM_API_KEY"

The key does not protect the whole server. The vLLM security guide (opens in a new tab) says --api-key only covers paths under /v1, /v2, /inference and /cohere. Other routes on the same port need no key, including /invocations (which reaches the same inference functions as /v1), /pause, /abort_requests, /score, /rerank, /tokenize, /health and /version. Anyone who can reach the port can use them.

Pre-launch security checklist

Copy this before exposing vLLM beyond your own machine:

• ☐To do: Set --api-key or VLLM_API_KEY, and keep the key out of shell history and logs.
• ☐To do: Put a reverse proxy (nginx, Envoy or a Kubernetes Gateway) in front and allowlist only the routes you use, such as /v1/chat/completions and /v1/models.
• ☐To do: Block every other route at the proxy, especially /invocations, /pause, /abort_requests and /update_weights.
• ☐To do: Firewall the host so only the API port is reachable, and keep torch.distributed and KV-transfer ports on trusted networks.
• ☐To do: Never set VLLM_SERVER_DEV_MODE=1 in production; it exposes /collective_rpc and cache-reset routes.
• ☐To do: Leave profiler endpoints and --enable-tokenizer-info-endpoint off unless you need them.
• ☐To do: Bind explicitly with --host (for example --host 127.0.0.1 behind a local proxy), since the CLI reference lists no default host.
• ☐To do: Terminate TLS at the proxy, or use --ssl-keyfile and --ssl-certfile on vLLM itself.
• ☐To do: Narrow --allowed-origins from its default of ['*'] if browsers call the API directly.