Skip to content
HUIAHUB

Topic

AI observability & evaluation

Tracing, evals, guardrails and monitoring for AI apps: the platforms, open-source libraries and guides for measuring what a model does.

AI observability and evaluation is the practice of recording what an AI application does, such as prompts, outputs, latency and token costs, and testing those outputs against expected results. To start with AI observability, follow the guide to tracing an LLM app with Langfuse, then the guide to testing prompts and LLM outputs with promptfoo, which ends with a CI step that fails a build on regressions.

Langfuse also has a tool page, which describes it as open-source tracing, evaluation and prompt management for LLM apps and agents, run as a cloud service or self-hosted. Other tool pages cover Agent Activity, a Mac app that shows what coding agents do in Xcode, and Polylane, which watches production logs, metrics and traces and opens pull requests with fixes for a team to review.

The listed talk is for engineers who run LLM apps in production and covers which cost, safety and quality metrics to track beyond latency and error rates. Use the guides for hands-on setup and the tool cards to compare pricing labels and API access.