Skip to content
HUIAHUB

jundot/omlx

omlx

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

Metrics updated Established
View on GitHub (opens in a new tab)Homepage / demo

GitHub metrics

Stars
23K
growth after 7 days of data
Forks
2K
30-day growth
—
Open issues
596

The chart appears once the repository has been checked on two or more days.

What it does

oMLX is a local LLM inference server for Macs with Apple Silicon, built on MLX and managed from a menu bar app. It exposes OpenAI-compatible and Anthropic-compatible endpoints, so existing clients can connect to http://localhost:8000/v1 without code changes.

Its main idea is a two-tier KV cache. Frequently used cache blocks stay in RAM, and when that hot tier fills up, blocks are offloaded to the SSD in safetensors format so they can be reused instead of recomputed. The author pitches this at long coding sessions with tools such as Claude Code. Concurrent requests go through continuous batching using mlx-lm's BatchGenerator.

The server loads text LLMs, vision-language models, OCR models, embeddings and rerankers. Several models can stay loaded at once, with least-recently-used eviction, pinning and a per-model idle timeout. A web admin dashboard at /admin covers monitoring, model management, chat and benchmarks. Source builds also include an experimental mode that splits one model across several Macs. You can install it as a .dmg from GitHub Releases, through Homebrew, or from source, and it is licensed under Apache-2.0.

Why it matters

Coding agents resend long contexts on every turn, and recomputing that prefix on a laptop is slow. oMLX keeps KV cache blocks in RAM and spills them to SSD, so earlier context stays reusable across requests. It pairs that with multi-model controls (pinning, per-model TTL, LRU eviction) and a native menu bar app. The README says the author built it because other LLM servers forced a choice between convenience and control.

Main features

  • Tiered KV cache with a RAM hot tier and an SSD cold tier
  • Continuous batching of concurrent requests through mlx-lm's BatchGenerator
  • OpenAI- and Anthropic-compatible APIs for chat, embeddings and messages
  • Serves text LLMs, VLMs, OCR models, embeddings and rerankers
  • Multi-model serving with LRU eviction, pinning and per-model idle TTL
  • Admin dashboard at `/admin` for monitoring, chat, benchmarks and model downloads
  • Tool calling with JSON schema validation and optional MCP support
  • Native menu bar app with in-app auto-update

Who it's for

  • Mac users on Apple Silicon who want local models behind an OpenAI- or Anthropic-compatible API
  • Developers connecting coding agents such as Codex, OpenCode or Copilot to a local model
  • People who prefer a GUI for loading, pinning and monitoring several models

Strengths

  • SSD-backed KV cache lets earlier context be reused across requests
  • Runs several models side by side with pinning, TTLs and memory limits
  • Three install paths (DMG, Homebrew, source) and a guided Welcome screen
  • The admin dashboard vendors its CDN dependencies, so it works offline

Limitations

  • Apple Silicon and macOS 15 or later only
  • GLM-5.2, MiniMax M3 and Qwen3.5 fall back to much slower paths without the native custom kernels
  • Multi-Mac inference is labeled experimental and is limited to source builds
  • The server refuses non-loopback binds without a main API key, so LAN use needs key setup

Getting started

The macOS app ships as a .dmg on GitHub Releases. Homebrew uses the project's own tap; the custom-kernel option needs full Xcode, and brew services keeps the server running in the background.

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx --with-custom-kernel
brew services start omlx

Requirements: macOS 15.0+ (Sequoia), Python 3.11–3.13 and Apple Silicon (M1/M2/M3/M4/M5); full Xcode with the Metal toolchain to build the native custom kernels.

Always follow the official instructions in the README on GitHub. This overview is our summary, not a copy.

Sources