What Makes Open Models Fast in Production — Sujee Maniyam, Nebius

Recorded at the AI Engineer World's Fair 2026 in San Francisco, this talk has two speakers from Nebius, although the title names only one. Dylan Bristot is senior AI product marketing manager for Nebius Token Factory; Sujee Maniyam is an AI developer advocate. The starting premise is that open models have nearly caught up with proprietary ones, and that serving them fast and cheaply is now the hard part.

According to the description, Dylan covers the business side. After the intro and a framing chapter at 0:12, the talk introduces Nebius (0:52) and Token Factory (2:27). At 2:42 it weighs closed APIs against self-hosting, including what the choice means for cost and lock-in, and at 4:06 it describes a loop of inference, data, post-training and deployment. Chapters at 6:21 and 8:06 cover what the platform handles for customers and the step from running models to improving them. Much of the first ten minutes is about Nebius's own product.

Sujee's part works through the optimizations behind fast inference, one layer at a time. After chapters on scaling open models (10:01), how good open models are (10:26) and teacher and student models (11:26), the hardware layer comes up at 11:46 with NVFP4 on recent NVIDIA hardware. Choosing a serving engine follows at 12:26.

Cache-aware routing, at 12:55, is set against naive load balancing for LLMs. Speculative decoding starts at 14:24, and at 15:29 the talk covers training custom draft models on your own traffic. KV cache offloading is at 15:59, disaggregated prefill and decode at 17:33, and finding the quantization sweet spot at 18:13.

A recap at 18:43 is followed by a closing pitch to try Token Factory at 19:22. The description links Token Factory, its quickstart docs and the Nebius site; pricing is not covered in the sources.

The second half is the part to jump to for engineers who run their own inference and want a list of techniques to look into. Skip to 10:01 if the vendor background is not useful to you.