Three Questions Every Agent Engineer Should Be Able To Answer

Jake Broekhuizen, from LangChain Labs, recorded this 14-minute talk during LangChain Presents: The Learning Loop, an evening event about continual learning hosted in San Francisco. His starting point (0:00) is the idea that a team should control the intelligence inside its product. At 0:38 he breaks that down into three checks: can you locate the failures, can you trigger them again on demand, and can you change something so they go away.

Locating failures comes first, from 1:30, by searching through traces from real usage. Using a large frontier model as the grader gets costly once traffic grows, so at 2:27 he describes training a smaller, fine-tuned grader to watch agent behavior in its place.

Reproduction is the subject from 4:10. Evals that only compare an input with an output fall short here. Starting at 5:25 he talks about specifying a simulated world plus a set of tasks that resemble real traffic. The worked example at 6:20 is the go-to-market agent LangChain runs internally. At 7:45 he discusses making those tasks difficult and resistant to reward hacking, and at 8:41 he shows how logged usage can feed the creation of new tasks and simulated worlds. From 9:15 he makes the case that product teams hold an advantage over frontier labs, because they observe how actual users work with their agents.

Fixes arrive at 11:08, applied both to the code wrapped around the model and to the model itself. The closing chapter at 12:34 looks at keeping up when the underlying models shift and user behavior changes.

Two links in the description add context: LangChain's blog post about building its GTM agent, and the LangSmith Observability product page.