Skip to content
HUIAHUB

Topic

Inference & model serving

Fast inference engines and model servers, with throughput, batching and deployment guides for serving models on one machine or a cluster.

Inference serving is the work of running a trained model behind an API so applications can send requests and get responses at a speed and cost that fit the use. To start with inference serving, follow the guide to serving an open model with vLLM behind an OpenAI-compatible API, or read the comparison of llama.cpp, Ollama and LM Studio if the model will run on one machine.

The tool pages cover Ollama, an open-source tool and API for running open models on your own machine or on Ollama's cloud, and Replicate, a cloud API for running, fine-tuning and deploying models with usage-based billing. Clef, from Cloudflare, is a set of open-weight decision models hosted on Workers AI.

The video is a talk for engineers serving open LLMs, with a layer-by-layer list of inference optimizations from NVFP4 to speculative decoding. Self-host with vLLM or Ollama when you need control over hardware and data, and compare the pricing labels on the hosted tool cards when you would rather not run servers.