RAG Evaluation: Faithfulness, Context Precision and Recall with RAGAS

TL;DR — RAG (Retrieval-Augmented Generation) systems have become the default architecture for enterprise AI projects; roughly 70% of engineering teams are shipping RAG in some form. But there is a serious gap between "it works" and "it's trustworthy." In this piece I explain how to read the four core RAGAS metrics — faithfulness, answer relevancy, context precision, context recall — together as a diagnostic panel, how the RAGAS/TruLens/DeepEval tools differ, how to interpret the ~0.85 "good" and ~0.70 "serious problem" thresholds, the emerging fifth dimension called "context trustworthiness" (ownership, freshness, source lineage), and how to build a golden test set, avoid LLM-as-judge pitfalls, integrate evaluation into CI, and navigate the Turkish enterprise/KVKK context. The goal is simple: turn RAG evaluation from a one-time check into a continuous discipline.

Over the past year and a half, in nearly every company I've consulted or trained, I've watched the same scene play out: a team takes their internal document repository — procedures, product catalog, or customer service knowledge base — embeds it into a vector database, puts an LLM on top, and declares "our chatbot is ready." Demo day impresses everyone. Three weeks later, the picture looks completely different: users start bringing complaints like "this answer is wrong," "I don't understand where it got this information," or "it's still quoting the policy that changed last month."

This isn't because RAG is a bad architecture. The problem is that most teams run their RAG system through a single "does it work" test and then push it to production. But RAG is a different animal from classic software: the input-output relationship isn't deterministic, the retrieval layer can degrade over time (documents get updated, embedding models change, the index grows), and the generation layer has a tendency to "make things up" when something isn't in its context. This is why evaluating RAG continuously, systematically, and measurably — not as a one-time quality check but as a lasting engineering discipline — becomes a competitive advantage. The gap between teams that do this and teams that don't becomes strikingly visible after about six months.

In this piece, centering on RAGAS — the most widely used evaluation framework in the field — I'll walk through the four core metrics, how they should be read together, what the various tools offer, and how to build an evaluation pipeline in practice. This isn't a theoretical article; it's a practical guide distilled from problems I've encountered on the ground.

In traditional software testing, the expected output is well-defined: you give the function X, you expect Y, and if they don't match, the test fails red. In RAG systems, there's no such clean match. The same question can receive two different answers that are both correct. Worse, errors can surface at two distinct layers:

1. The retrieval layer — can the system find the right documents?
2. The generation layer — can it produce a correct, coherent answer from the documents it found?

When a RAG system gives a bad answer, if you can't tell whether the root cause is retrieval or generation, you don't know what to fix either. Do you swap the embedding model, shrink the chunk size, or improve the prompt? Being able to make this distinction is the crux of RAG evaluation. RAGAS's (Retrieval Augmented Generation Assessment) design philosophy is built exactly on this: measure retrieval quality and generation quality separately, then interpret them together.

I want to stress one point here in particular: reading these four metrics in isolation, one by one, will mislead you. A system with high faithfulness but low context recall means "it doesn't hallucinate, but it gives incomplete information" — it may be telling the user half the truth. A system with high context precision but low answer relevancy may have found the right documents but drifted off-topic instead of answering the question. These four metrics need to be read together, like a diagnostic dashboard.

Faithfulness — catches hallucination

Faithfulness measures how many of the claims in the generated answer are actually supported by the retrieved context. In practice it works like this: the answer is first broken down into atomic claims, then each claim is checked against the context to ask "can this claim be inferred from the context?" The ratio of supported claims to total claims gives the faithfulness score.

The failure mode this metric targets is clear: hallucination. An LLM can pull information that isn't in its available context from its own parametric memory (pretraining data) and insert it into the answer. Sometimes this information might even be correct — but the entire point of a RAG system was to make the answer verifiable and grounded in sources. A claim not present in the context, even if true, undermines the system's reliability because it can't be audited.

A typical example I've seen in the field: an HR chatbot pulls "annual leave is 14 days" from the leave policy document but adds a sentence to its answer like "sick leave is also included within this period" — this second sentence isn't in the context; it's the model's own generalization. The faithfulness metric exists precisely to catch this.

Answer relevancy — catches off-topic drift

Answer relevancy measures how directly the generated answer addresses the actual question asked. Technically, several possible questions are generated backward from the answer (using an LLM), and the semantic similarity (embedding cosine similarity) between these generated questions and the original user question is computed. The more the answer produces questions that "match" the original, the more relevant it's considered.

The failure mode this metric targets is: the system may have found correct information and stayed faithful to the context, yet still failed to answer the question. For example, a user asks "what is the remote work policy at the Izmir office?" and the system produces a long, accurate but not Izmir-specific text describing the company's general remote work policy. Faithful to the context, no hallucination — but no clear answer to what the user actually asked. Answer relevancy comes out low.

It's important to separate this from faithfulness because they're independent axes: a combination of high faithfulness and low answer relevancy points to systems that produce "correct but irrelevant" answers — usually indicating a problem in prompt design or the query understanding layer.

Context precision — catches retrieval noise

Context precision measures how many of the retrieved context chunks are actually relevant to the question. RAGAS typically uses a rank-sensitive calculation: chunks that are more relevant are rewarded for being ranked higher (similar to the logic of precision@k in information retrieval).

This metric measures pure retrieval quality; it has nothing to do with generation. The failure mode it targets: noisy retrieval. The system retrieves 5 document chunks, but only 1-2 of them are actually relevant to the question, and the rest are irrelevant or partially relevant. This creates two risks at once: (1) because the context window fills up with unnecessary information, the model may "lose" the truly important piece, and (2) irrelevant chunks can distract the model's attention and lead it to a faulty synthesis.

Low context precision usually stems from: chunk size chosen too large or too small, an embedding model that doesn't fit the domain well, or a missing re-ranking layer during retrieval.

Context recall — catches incomplete retrieval

Context recall measures how much of the information necessary to produce the correct answer is actually present in the retrieved context. To calculate this, a reference (ground truth) answer is typically used: the claims in the reference answer are taken one by one, and for each, the question is asked "is this claim supported by the retrieved context?" The ratio of supported claims gives the context recall score.

The failure mode it targets is the mirror image of context precision: incomplete retrieval. The system's retrieved chunks might all be relevant (high precision), but it may have failed to retrieve a critical piece of information needed for the answer (low recall). For example, in a contract query, if an exception to the relevant clause appears in a separate section and retrieval never pulls that section, the model has no way of knowing about that exception — because it simply doesn't have it. This looks like an "innocent" error from a faithfulness standpoint (the model isn't making things up), but from a business-outcome standpoint it leads to an incomplete and potentially wrong answer.

Context recall is directly tied to the system's index, chunking strategy, and top-k setting (how many chunks are retrieved). Because this metric requires a reference answer, it can generally only be measured on a curated "golden set" — not on live production traffic.

These four metrics need to be read together, not one by one, like a diagnostic dashboard. The table below summarizes which combination points to which root cause — this is the practical framework I reach for most often when working with teams in the field:

The real value of this table is this: because retrieval quality (context precision + context recall) and generation quality (faithfulness + answer relevancy) are measured as mathematically distinct axes, when you see a drop, you know where to look. Without this distinction, when you're faced with a vague complaint like "the answer was bad," guessing whether to change the embedding model, revise the prompt, or adjust chunk size becomes close to a blind shot. The biggest time-waster I've seen on teams I've consulted is skipping this distinction and jumping straight into "let's change a bit of everything and try again" mode.

RAGAS metrics produce scores normalized between 0 and 1. Based on what I've seen in the field and what's generally accepted, the interpretation ranges look like this:

I'd recommend treating these thresholds as a compass rather than absolute truth. For instance, for a system operating in a high-risk domain like legal or healthcare, even 0.85 may not be sufficient; whereas for a low-risk internal knowledge assistant, something around 0.80 may be acceptable. What matters is establishing your own baseline for your domain and tracking how it changes over time. A score isn't meaningful as a single number on its own — it's meaningful as a trend: a faithfulness score that was 0.88 last month dropping to 0.78 this month is an alarm bell regardless of the absolute value — the source documents probably changed, the embedding model got updated, or the model provider silently pushed a version update in the background.

One more caveat: these scores are produced via the LLM-as-judge method (which I'll detail in the next section), meaning they carry an inherent margin of uncertainty. It's normal to see a few points of variance when you run the same test set twice. For this reason, I'd recommend relying on the average of multiple runs and the trend line rather than a single measurement.

Now I want to talk about a problem I'm encountering increasingly often in the field — one that the classic RAGAS quartet doesn't catch. A system can score 0.95 on faithfulness — meaning every claim it produces is fully supported by the context it retrieved — and still give a completely wrong, even harmful, answer from a business standpoint. How?

Say your system stays entirely faithful to the context it retrieves, never making anything up. But the context it retrieves is a price list that was updated six months ago but never re-indexed. Or it surfaces the wrong one out of two conflicting procedure documents from different departments. Or it treats a former employee's personal notes — never approved as an official source — as a "source." In all three cases the faithfulness score comes out high, because the model genuinely stayed faithful to what it found. But the answer is wrong, because the context itself is not trustworthy.

I call this the "fifth dimension": context trustworthiness. The classic four metrics measure "the relationship between the retrieved context and the answer," but none of them measure how trustworthy the source of the context itself is. This breaks down into three sub-dimensions:

• Ownership: Who approved this document, who is responsible for it? Is it an official source, or a user's personal notes?
• Freshness: When was this information last updated? Is an old, revoked version still sitting in the system?
• Lineage: Which original, canonical source was this information derived from? If multiple versions exist, which one is considered "correct"?

The most common scenario where I see these three problems in enterprise RAG projects is this: a company takes a messy pile of documents accumulated over years — SharePoint, Confluence, email attachments, old presentations — and embeds it as-is into a vector database. No curation, no check of "is this document current, who approved it" is ever done. The result: the system works technically flawlessly (faithfulness, precision, recall are all high), but gives wrong, outdated answers in the real world — because the source it's fed from is already untrustworthy.

The practical fix for this is to add context trustworthiness checks to your evaluation pipeline:

1. Mandatory metadata: Add metadata fields such as owning unit, last update date, and approval status to every document chunk, and use these for filtering/weighting during retrieval.
2. Staleness warnings: Automatically flag documents older than a certain date as "needs review" and lower their retrieval priority.
3. Canonical source matching: When multiple documents exist on the same topic, determine which one is the "single source of truth" and archive or exclude conflicting versions from retrieval.
4. Periodic document audits: Regularly confirm through human review that indexed content is still accurate and owned by someone.

Measuring this fifth dimension isn't as standardized yet as the classic RAGAS metrics — there's no single, universally accepted metric for it in the industry yet. But my recommendation is to define a simple "trust score" for your own organization: for instance, a metric that tracks what percentage of retrieved chunks come from documents updated and owned within the last 6 months. This can be added as a fifth column to your RAGAS panel, and it becomes as critical as faithfulness — especially in regulated, high-risk enterprise use cases.

There are a few mature open-source frameworks on the market for RAG evaluation. I've used all three on different projects; each has its own strengths and weaknesses.

RAGAS, as the name suggests, was born specifically for RAG evaluation. It offers metrics like faithfulness, answer relevancy, context precision, and context recall as ready-made functions, and integrates easily with popular RAG frameworks like LangChain and LlamaIndex. Lightweight, quick to set up, well documented. It's my first choice especially for teams wanting a "quick start" and a "deep, RAG-specific metric set."

TruLens offers a broader "LLM application observability" approach. It also supports RAG metrics, but its real strength is a flexible structure they call "feedback functions," which lets you define your own custom metrics and monitor them over live traffic through dashboards. It's better suited for teams that want to continuously monitor a production system and see trend graphs over time.

DeepEval is a library built on a "pytest for LLMs" philosophy. It also includes RAG metrics, but its scope is broader: it offers general LLM output evaluation, safety tests (bias, toxicity), and custom metric definition. Teams accustomed to a test-driven development culture find it very natural to integrate DeepEval into their CI/CD pipelines because its syntax genuinely resembles pytest.

A brief comparison:

In practice these three aren't mutually exclusive; many teams use RAGAS for core metric computation, DeepEval for automated regression testing in CI/CD, and TruLens for continuous monitoring in production, all together. My recommendation is to start small: first set up offline evaluation on a golden set with RAGAS, then add CI integration and live monitoring layers as the system matures.

No metric can be better than the test set it's run against. The most neglected but most critical step in RAG evaluation is building a good "golden set" — a curated test set made up of questions, expected context, and reference answers.

A good golden set should include the following elements:

• Realistic questions: Questions compiled from real user logs, support tickets, and frequently asked questions. Rather than synthetic, "clean" questions, it should include examples reflecting how users actually write (typos, missing context, cramming multiple questions into one sentence).
• Difficulty variety: Alongside simple single-document questions, it should also include questions that require synthesizing multiple documents, or ones whose correct answer should be "no, there is no such information" (negative examples, which test whether the system knows when to say "I don't know").
• Reference answers: Answers approved by domain experts and considered correct. This is essential, in particular, for calculating context recall.
• Edge cases: Boundary situations such as documents with conflicting information, outdated content, or questions containing highly technical jargon.

If you ask how many questions is enough, giving a precise number would be misleading, but here's my practical experience: starting with a few dozen questions (around 30-50) and expanding to hundreds over time so as to cover the system's different usage scenarios (department, topic, question type) is a reasonable path. What matters isn't the number, but representativeness — whether it truly reflects the distribution of questions you'll encounter in production.

Don't build the golden set once and shelve it. As source documents change, user behavior evolves, and new features get added, update the golden set too. I recommend that teams I advise treat the golden set as a "living document," with a product manager or domain expert regularly (say, monthly) reviewing it and adding new examples.

Most tools like RAGAS, TruLens, and DeepEval use an LLM as a "judge" to compute metrics — meaning, to compute faithfulness, you ask another (or the same) LLM "can this claim be inferred from this context?" This approach is powerful because it's scalable and much cheaper than human evaluation. But it has some pitfalls, and using it without being aware of them can lead to misleading results:

The judge model's own biases. LLM-as-judge carries the biases of the data it was trained on. For example, some studies show that judge models systematically tend to rate longer answers as "better" (length bias). A short but accurate answer might score lower than a long but loosely written one.

Self-evaluation bias when the judge and the generator share the same model family. If the model that generates the answer and the model that evaluates it come from the same family/provider (say, both are models from the same company), the model may develop a tendency to find its own style "more natural" and systematically give it higher scores. Where possible, choosing the generator and judge models from different providers gives a healthier signal.

Lack of determinism. LLM-based judges may not give the exact same score to the exact same input every single time. For this reason, rather than over-relying on a single measurement, it's helpful to look at the average of multiple runs and, where possible, keep the temperature parameter low.

Missing subtle nuances. LLM judges don't always catch answers that look superficially "correct" but are actually subtly wrong — especially when it comes to numerical calculations, date/time logic, or domain-specific technical terms. In these kinds of critical areas, the healthiest approach is to treat LLM-as-judge scores not as absolute truth but as a pre-filter, routing examples that score low or look risky to human review.

In short: LLM-as-judge should be seen as a tool that scales human evaluation, not one that replaces it. I recommend to teams: use LLM-as-judge for broad-scale, continuous monitoring, but at regular intervals (say, quarterly), also have a subset of the golden set evaluated by human experts and check the correlation between the LLM judge's scores and the human scores. If the correlation is low, revisit the judge prompt or the judge model.

The real value of RAG evaluation emerges when you make it a natural part of the development loop. The most practical way to do this is to integrate tools like RAGAS or DeepEval into your CI/CD pipeline — meaning every code change, every prompt update, every embedding model change is automatically run against the golden set to produce scores, and if any metric falls below the threshold, the build fails.

A simple example flow might look like this:

# .github/workflows/rag-eval.yml (conceptual example)
name: RAG Evaluation
on: [pull_request]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run RAGAS evaluation against the golden set
        run: python scripts/run_ragas_eval.py --dataset golden_set.jsonl
      - name: Threshold check
        run: |
          python scripts/check_thresholds.py \
            --faithfulness-min 0.85 \
            --context-recall-min 0.80 \
            --fail-on-regression

The biggest benefit this kind of setup brings is catching regressions early. An engineer changing a prompt to "improve" it might unknowingly lower faithfulness. A team that changes chunk size might cause a silent drop in context recall. Without automated evaluation, these changes are typically noticed weeks later, through user complaints. An eval set integrated into CI catches these kinds of regressions before they're merged.

Practical recommendations:

• Set threshold values gradually. Start with loose thresholds (say, 0.75) and tighten them (toward 0.85) as the system matures.
• Define separate thresholds for each metric. Instead of a single "average score," set a separate minimum for each metric like faithfulness and context recall — because an improvement in one metric can mask deterioration in another.
• Keep costs in check. LLM-as-judge calls can be expensive; instead of running the entire golden set on every PR, using a representative subset and running the full set before merging to the main branch can be a reasonable balance.
• Make results visible. Don't leave scores buried in CI logs; push them to a dashboard (tools like TruLens naturally offer this) and track them as a trend.

To make this concrete, let me walk through a scenario I encounter frequently in the field. Picture a Turkish bank or financial institution: a RAG assistant is built to help customer service representatives, containing internal procedures, product terms, campaign conditions, and regulatory documents. The representative asks a question ("what is the deadline for a credit card statement dispute?"), and the system generates an answer from the relevant procedure.

In this scenario, all four metrics are critical, but for different reasons:

• Faithfulness is critical because the representative may pass whatever the system says directly on to the customer; a hallucination could result in the customer receiving wrong information, or even in issues with regulatory bodies (like BRSA/BDDK, CMB/SPK).
• Context recall is critical because in regulation-driven processes (like dispute deadlines, interest rate change notices), missing an exception or a recent change leads to both customer harm and institutional risk.
• Context trustworthiness (the fifth dimension) is especially critical here: in the financial sector, regulations change frequently, and campaign conditions are seasonal. If the system presents last quarter's campaign conditions as current, it may have given a technically "faithful" answer, yet the business outcome is still wrong.

For a system like this, my recommendation is to build the golden set not just from general questions but also to include "is this information current" tests derived from regularly updated regulatory and campaign documents. It's also reasonable to keep the thresholds for faithfulness and context recall stricter than in general use cases (say, above 0.90), because the cost of error is high.

The KVKK and data governance dimension

When building a golden test set, especially in sensitive-data domains like banking, healthcare, or HR, there are a few points to keep in mind under KVKK (Turkey's Personal Data Protection Law):

• Building a test set with real customer data is risky. When deriving the golden set from real production logs, personal data (names, national ID numbers, account numbers, phone numbers, and so on) needs to be anonymized or masked. Even in a test/eval environment, using real personal data counts as a "processing" activity under KVKK and is subject to principles like purpose limitation and data minimization.
• Supplement with synthetic data. Using realistic but synthetic (generated) test scenarios as much as possible both reduces KVKK risk and makes it easier to enrich the test set with whatever edge cases you want.
• Auditability. The evaluation process itself needs to be auditable — keeping records of which golden set was used, which thresholds were applied, and which version was deployed to production matters both for internal audit and for any potential regulatory review. Logging your RAG system's faithfulness and context trustworthiness scores over time gives you a concrete answer to the question "how did we validate this system?"

This isn't just a compliance formality; it's also a way of building trust. I always tell my enterprise clients: a metric dashboard showing how well a RAG system is performing isn't just for the engineering team — it's an assurance tool for risk, compliance, and senior leadership as well. Saying "our system doesn't hallucinate" is far less convincing and defensible than saying "our faithfulness score has averaged 0.91 over the last three months, and our context trustworthiness check confirms document freshness every month."

I'd recommend building RAG evaluation not as a one-time delivery criterion, but as a continuous habit embedded in the product's lifecycle. The large majority of engineering teams are now shipping RAG in some form; this means that building RAG is no longer a differentiator, but operating it in a reliable and measurable way has become one. From what I've seen in the field, teams that monitor faithfulness, answer relevancy, context precision, and context recall together as a panel, embed this into CI/CD, keep their golden set alive, and don't overlook context trustworthiness clearly pull ahead within six months to a year: fewer user complaints, faster error diagnosis, and — most importantly — users who trust their systems.

Building this discipline may look like extra work at first — preparing a golden set, setting thresholds, integrating into CI all take time. But the cost of not doing it is much higher: a system that silently degrades in production, loss of trust, and in the worst case, in a regulated sector, a serious compliance problem. I tell every team I advise the same thing: building RAG is the easy part; keeping it trustworthy is the real work.