In-Context Retrieval with Siddharth Gollapudi - Weaviate Podcast #146!

Weaviate Podcast #146 is a 56-minute conversation with Siddharth Gollapudi of UC Berkeley about his research on in-context retrieval, the subject of his paper "Can Language Models Actually Retrieve In-Context?" The idea he tests is to drop the embedding model and the ANN index, place every document in the LLM's context at once, and let the model's attention do the finding.

Once the host welcomes him, the 1:21 chapter sets in-context retrieval apart from two neighbors. Generative retrieval bakes documents into the weights, so each change to the collection means more training. Gollapudi treats listwise re-ranking as a simpler version of the first-stage problem, and he says the lost-in-the-middle effect has faded for current long-context models. From 8:16 the discussion turns to the Drowning in Documents work on what happens when reranker inference is scaled up.

BlockSearch takes the chapter from 13:05. It is a 0.6B parameter model, and its training mix combined MS MARCO (filtered with RLHN) and several BEIR sets. Per the episode notes, getting to 500 documents went smoothly, while 2,500, 5,000 and 10,000 documents (close to a million tokens) is where accuracy fell apart. Gollapudi gives his reasons for preferring random document identifiers over position-based ones, and describes an on-policy loss that trains the model on its own mistakes when similar distractor documents lead it astray. The notes also cover attention dilution, where a growing corpus spreads the softmax too thin, and two remedies: scaling temperature with input length, and sparse attention that discards unrelated documents first. With both, BlockSearch roughly matches dense retrieval at a million tokens.

The 28:21 chapter looks at vector databases for LLM inference and asks whether attention could get a sublinear, ANN-like form that still trains end to end, citing ReFrag and ColBERT late interaction. Timelines for in-context retrievers come at 40:46. At 46:54 the pair debate whether LLMs will eat databases, with references to "No More Free Lunch" and recursive language models. The last chapter, at 51:38, covers the research directions Gollapudi is most interested in, among them relevance signals learned without supervision.