Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

Marina Petzel, GenAI advocate at Datadog, gave this talk at the AI Engineer World's Fair 2026 in San Francisco. Her point is that the classic golden signals (latency, errors, traffic and saturation) tell you an app is up but not whether its answers are right. She keeps them and adds three groups of metrics for generative AI apps: cost, safety and quality.

After the intro, the chapter at 0:12 explains why golden signals are no longer enough. From 1:12 she covers how GenAI apps differ; the description counts four ways they break traditional monitoring assumptions. New attack vectors (2:22) and the subjectivity of quality (2:47) are two of the chapter headings in this part.

Cost monitoring starts at 3:51. At 4:56 she names three hidden cost drivers: token creep, model drift and uncached calls. From 6:55 the advice is to tag everything for cost attribution, using what the description calls a four-level tagging strategy.

Safety metrics begin at 9:15. The chapter at 9:55 covers prompt injection, PII leakage, toxicity and jailbreaks, and the description says she gives target thresholds for these metrics. The thresholds themselves are not listed in the sources.

Quality monitoring follows at 12:09, with five quality metrics from 13:19: hallucination rate, relevance, user satisfaction, completeness and RAG retrieval quality. At 15:43 she puts the three groups together.

The last chapter, at 16:38, is about Datadog's Agent Observability product, and the description links it along with Datadog's LLM Observability documentation.

The talk suits SREs and engineers who already run LLM features and want a checklist of what to measure beyond uptime.