How to evaluate a RAG pipeline with Ragas: metrics, a test set and a first run

RAG evaluation with Ragas splits the score into two halves. Context precision and context recall tell you whether the retriever found and ranked the right chunks, while faithfulness, answer relevancy and factual correctness tell you whether the model answered from them. You need a small set of questions with reference answers, a judge model and one scored run per pipeline version.

What can fail in a RAG pipeline: retrieval or generation

A RAG answer can go wrong in two places. The retriever can miss the chunk that holds the answer, or return it below several irrelevant chunks. The generator can then ignore the context, add claims the context never made, or drift away from the question.

The two failures need different fixes. A retrieval miss points at chunking, the embedding model, the number of chunks returned or a missing reranker. A generation miss points at the prompt or the model. A single "answer quality" score hides which half broke, so you end up changing chunking when the prompt was the problem.

The paper that introduced Ragas (arXiv 2309.15217 (opens in a new tab), by Shahul Es, Jithin James, Luis Espinosa-Anke and Steven Schockaert) frames the same split. It scores the retriever's ability to find relevant passages, the LLM's faithfulness to those passages and the quality of the generated text as separate dimensions.

How to install Ragas

The Ragas repository (opens in a new tab) is maintained by vibrantlabsai under the Apache-2.0 license. The package on PyPI (opens in a new tab) is at version 0.4.3, released January 13, 2026, and requires Python 3.9 or newer.

pip install ragas

For a ready-made project, the quickstart command scaffolds a RAG evaluation template with rag.py, evals.py and an evals/ folder for datasets, experiment results and logs:

ragas quickstart rag_eval
cd rag_eval

The template calls OpenAI by default and reads OPENAI_API_KEY. The quickstart docs show how to swap the judge for Anthropic, Google Gemini or a local model behind an OpenAI-compatible endpoint, such as Ollama. Ragas collects anonymized usage data, and setting RAGAS_DO_NOT_TRACK to true turns that off.

How to build a small evaluation set

Each Ragas sample for a RAG pipeline uses four fields:

• user_input: the question.
• retrieved_contexts: the list of chunks your retriever returned.
• response: the answer your pipeline generated.
• reference: the answer you expect, written or checked by a person.

You write user_input and reference. Your pipeline fills in retrieved_contexts and response when you run it, so the same questions can be replayed against every pipeline version.

Pick the questions

Start with a few dozen questions you can read through by hand in one sitting. Take them from real user traffic or support tickets where you can. Cover the cases that worry you: questions whose answer sits in one chunk, questions that need two chunks, and questions your corpus cannot answer at all.

Keep each reference answer short and factual. Several metrics break the reference into claims, so a padded reference produces padded claims.

Generate more with the test set generator

Ragas can draft extra questions from your documents. The TestsetGenerator class builds a knowledge graph from LangChain or LlamaIndex documents and writes questions from it:

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
testset = generator.generate_with_langchain_docs(docs, testset_size=10)
testset.to_pandas()

The Ragas docs suggest reviewing the generated set and keeping only the queries that fit. Treat generated questions as drafts, since a model wrote both the question and its reference.

A sample row

The row below is illustrative. Replace it with your own data.

{
  "user_input": "How many days does a customer have to request a refund?",
  "retrieved_contexts": [
    "Refunds can be requested within 30 days of purchase.",
    "Shipping usually takes 3 to 5 business days."
  ],
  "response": "You can request a refund within 30 days of purchase.",
  "reference": "Customers can request a refund up to 30 days after purchase."
}

RAG evaluation with Ragas: which metric answers which question

The table maps the core metrics to the half of the pipeline they test. Class names are from ragas.metrics.collections, the API the Ragas docs recommend for new projects. The full list, including agent, SQL and summarization metrics, is on the Ragas available metrics page (opens in a new tab).

A few details from the metric pages change how you read the numbers:

• Context precision is order-sensitive. In the docs example, moving an irrelevant chunk from second to first place drops the score from about 1.0 to about 0.5.
• Context recall breaks the reference into claims and checks each against the retrieved chunks, so its quality depends on your reference answers.
• Faithfulness divides supported claims by total claims in the response. An Einstein example with one wrong date out of two claims scores 0.5.
• Answer relevancy generates 3 questions from the answer by default (the strictness parameter) and compares their embeddings with the original question. The score usually falls between 0 and 1, but cosine similarity does not guarantee it.
• FactualCorrectness defaults to f1 mode, with precision and recall as options.

How to run a first evaluation

The script below scores every row with five metrics. It uses llm_factory and embedding_factory as shown on the Ragas metric pages; rows is your list of dictionaries in the format above.

import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import (
    ContextPrecision, ContextRecall, Faithfulness, AnswerRelevancy, FactualCorrectness,
)

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)

precision = ContextPrecision(llm=llm)
recall = ContextRecall(llm=llm)
faithfulness = Faithfulness(llm=llm)
relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)
correctness = FactualCorrectness(llm=llm)

async def score(row):
    q, ctx, ans, ref = row["user_input"], row["retrieved_contexts"], row["response"], row["reference"]
    return {
        **row,
        "context_precision": (await precision.ascore(user_input=q, reference=ref, retrieved_contexts=ctx)).value,
        "context_recall": (await recall.ascore(user_input=q, retrieved_contexts=ctx, reference=ref)).value,
        "faithfulness": (await faithfulness.ascore(user_input=q, response=ans, retrieved_contexts=ctx)).value,
        "answer_relevancy": (await relevancy.ascore(user_input=q, response=ans)).value,
        "factual_correctness": (await correctness.ascore(response=ans, reference=ref)).value,
    }

async def main(rows):
    return await asyncio.gather(*(score(r) for r in rows))

The Ragas get-started guide still shows the older evaluate() call with EvaluationDataset, which returns averages such as {'context_recall': 1.0000, 'faithfulness': 0.8571, 'factual_correctness': 0.7280}. The API reference marks evaluate() as deprecated in favor of the @experiment decorator, and the metric pages say the legacy metric classes will be removed in version 1.0.

Reading the scores

Read the scores per row before you look at the averages. The patterns point to one half of the pipeline:

How to compare two pipeline versions

Change one thing between runs. The Ragas experimentation page lists "Isolate changes" as a principle, because two simultaneous changes make the score impossible to attribute. Keep the same test set and the same judge model for both runs.

Ragas has an @experiment decorator for this. Extra keyword arguments passed to arun reach your function, so one experiment can run under several settings:

from ragas import Dataset, experiment

@experiment()
async def rag_run(row, top_k: int):
    contexts = retrieve(row["user_input"], top_k=top_k)   # your retriever
    answer = generate(row["user_input"], contexts)        # your generator
    result = await faithfulness.ascore(
        user_input=row["user_input"], response=answer, retrieved_contexts=contexts
    )
    return {**row, "retrieved_contexts": contexts, "response": answer,
            "faithfulness": result.value, "top_k": top_k}

dataset = Dataset.load(name="rag_test", backend="local/csv", root_dir="./data")
baseline = await rag_run.arun(dataset, top_k=3)
candidate = await rag_run.arun(dataset, top_k=8)

Experiments save results as CSV files under experiments/, which makes a row-by-row diff easy. Rerun the baseline once without changes before you trust a small gain. The gap between two identical runs shows how far scores move with nothing changed, and any difference smaller than that gap is noise.

Where LLM-judged scores mislead

Most Ragas RAG metrics ask a judge model to split text into claims and label each one. That brings three limits.

Faithfulness is measured against the retrieved chunks, not against the truth. If retrieval returns an outdated policy and the model repeats it, faithfulness can still be 1.0. Pair it with context recall or factual correctness against a reference you trust.

The judge can also disagree with your experts. The Ragas guide on aligning an LLM judge (opens in a new tab) measured a gpt-4o-mini judge against 160 expert-labeled samples from EvalsBench. The baseline prompt agreed on 121 of 160 (75.6%), and all 39 disagreements were false positives, where the judge passed answers the experts failed. A revised prompt reached 139 of 160 (86.9%).

The lesson for your own runs is to read a sample of scored rows by hand before acting on an average. Log judge calls in a tracing tool such as Langfuse so you can see the claims the judge extracted and why a row scored low. If the judge keeps passing answers you would fail, fix its prompt or pick a stronger judge model before you compare pipelines with it.

When you have chunk IDs, you can skip the judge for retrieval. IDBasedContextPrecision and IDBasedContextRecall compare retrieved IDs with reference IDs, and NonLLMContextRecall uses string similarity. For faithfulness, FaithfulnesswithHHEM uses Vectara's HHEM-2.1-Open classifier for the claim check; it runs on CPU with a batch size of 10 by default.