How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs

Recorded at the AI Engineer World's Fair 2026 in San Francisco, this talk runs about 22 minutes and is split between two speakers. Daniel Kim, Head of Growth at Cerebras, takes the inference runtime. Michelle, a co-founder of Gimlet Labs, takes the system level.

After a short intro, the 0:33 chapter is about demand: the description cites Google going from 480 trillion to 3.2 quadrillion tokens a month. At 2:02 Kim explains autoregressive decoding and why it is memory-bound. He covers splitting models across GPUs at 2:37 and then, at 3:27, the trade-off between speed and throughput drawn as a Pareto frontier. Batching and continuous batching come at 4:27.

The KV cache gets three chapters in a row: its memory growth (5:22), the paged KV cache (6:11) and prompt caching (7:06). Kim compares the prefill and decode phases at 7:41 and closes his half with chunked prefill for long contexts at 8:36.

Michelle starts at 9:38 with the point that workloads differ once you look at the whole system. Prefill-decode disaggregation follows at 11:06. At 12:21 the talk compares memory bandwidth on Cerebras hardware with the H100, and at 13:05 it discusses Cerebras together with AWS Trainium. Gimlet's multi-silicon inference cloud is the topic at 13:39. Speculative decoding starts at 14:28, and attention-FFN disaggregation at 17:01.

The final stretch is about agents. From 18:25 the talk covers agents that write optimized kernels, and the last chapter at 20:16 widens that to agents for silicon, compilers and ops.