{"id":34,"date":"2026-06-03T06:30:00","date_gmt":"2026-06-03T06:30:00","guid":{"rendered":"https:\/\/regentis.ai\/blog\/?p=34"},"modified":"2026-09-07T12:47:04","modified_gmt":"2026-09-07T12:47:04","slug":"building-rag-is-not-enough-how-can-we-measure-the-reliability-of-generation","status":"publish","type":"post","link":"https:\/\/regentis.ai\/blog\/building-rag-is-not-enough-how-can-we-measure-the-reliability-of-generation\/","title":{"rendered":"Building RAG Is Not Enough: How Can We Measure the Reliability of Generation?"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

My previous articles<\/a> about RAG, hybrid approaches, and GraphRAG are already published. In those articles, I mostly focused on how information is retrieved, how context is built, and how company knowledge can be transferred to AI systems in a more meaningful way.<\/p>\n\n\n\n

Today, I want to focus on another side of this structure: the reliability of generation.<\/p>\n\n\n\n

Because in RAG systems, retrieving the right information is only half of the job. We also need to see whether the model actually makes correct use of that information.<\/p>\n\n\n\n

We cannot bring thousands of people together in one place and ask them to read and check all the data. So, a new evaluation problem appears here.<\/p>\n\n\n\n

How will we measure the answers produced by AI systems?<\/p>\n\n\n\n

How will we detect hallucinations?<\/p>\n\n\n\n

How will we notice answers that look correct but are not based on a source?<\/p>\n\n\n\n

LLM-as-a-Judge: using AI to evaluate AI output
<\/h2>\n\n\n\n

Today, one of the biggest crises created by artificial intelligence is reliability and reaching the correct result. There is a slightly ironic situation here. One of the most logical ways to solve this crisis is again using artificial intelligence as an evaluator.<\/p>\n\n\n\n

In the literature, this approach is known as LLM-as-a-Judge. The idea is that one LLM scores the output of another LLM against defined criteria. What makes LLM-as-a-Judge attractive is that it enables evaluation processes that are more scalable, more consistent, and cheaper to run. At the same time, the reliability, consistency, and bias risks of this approach must be handled with careful design.<\/p>\n\n\n\n

Why BLEU and ROUGE are not enough for LLM output
<\/h2>\n\n\n\n

Old traditional metrics are also not enough on their own here. Metrics like BLEU and ROUGE were developed for earlier NLP systems. Basically, they look at word or n-gram similarity between the generated text and the reference text. Microsoft\u2019s LLM evaluation guide also states that metrics like BLEU and ROUGE measure n-gram overlap between the reference text and the output. These metrics were used for a long time, especially in tasks like translation, summarization, and similar text generation tasks. But in LLM outputs, the problem is no longer only word similarity.<\/p>\n\n\n\n

An answer can be correct even if it does not repeat the exact words of the reference text. The reverse is also possible. It can share many words with the reference text and still express a wrong meaning. That is why relying only on mathematical similarity can fall short when we try to capture depth of meaning and contextual accuracy.<\/p>\n\n\n\n

RAG has two sides: retrieval and generation
<\/h2>\n\n\n\n

This problem becomes even clearer in RAG systems. Because here, the only thing we need to evaluate is not the final answer. There are two different sides of the process:<\/p>\n\n\n\n

The first side is whether the system brings the correct context.<\/p>\n\n\n\n

The second side is whether the model uses this context correctly.<\/p>\n\n\n\n

What is RAGAS?
<\/h2>\n\n\n\n

RAGAS becomes meaningful exactly at this point.<\/p>\n\n\n\n

RAGAS, which means Retrieval Augmented Generation Assessment, is a framework developed to evaluate RAG pipelines. In the RAGAS study, it is emphasized that RAG systems have two main components: retrieval and generation. The evaluation should not only look at the final answer. It should also look at the retrieval system\u2019s ability to find the relevant context, the LLM\u2019s ability to stay faithful to this context, and the quality of generation.<\/p>\n\n\n\n

For me, the important part of RAGAS is this:<\/p>\n\n\n\n

It does not look at the RAG system as one single whole. It does not evaluate from a very general point like \u201cis the answer good or bad?\u201d Instead, it divides the system into parts:<\/p>\n\n\n\n

    \n
  • Did the retriever bring the right information?<\/li>\n\n\n\n
  • Is the retrieved context really related to the question?<\/li>\n\n\n\n
  • Is the needed information inside the context?<\/li>\n\n\n\n
  • Did the model create the answer based on this context?<\/li>\n\n\n\n
  • Is the answer suitable for the real purpose of the question?<\/li>\n<\/ul>\n\n\n\n

    In enterprise AI, this separation matters even more. Because inside a company\u2019s internal systems, a wrong answer is not just poor user experience. It can lead to a wrong analysis, a wrong decision, misleading guidance, or a loss of trust.<\/p>\n\n\n\n

    The four RAGAS metrics
    <\/h2>\n\n\n\n

    There are four main metrics that stand out in RAGAS metrics:<\/p>\n\n\n\n

      \n
    • Faithfulness<\/li>\n\n\n\n
    • Answer Relevancy<\/li>\n\n\n\n
    • Context Precision<\/li>\n\n\n\n
    • Context Recall<\/li>\n<\/ul>\n\n\n\n

      RAGAS documentation uses these metrics to evaluate different parts of the RAG pipeline separately. Faithfulness and Answer Relevancy mostly look at the generation side, while Context Precision and Context Recall help us understand the retrieval side.<\/p>\n\n\n\n

      Faithfulness
      <\/h3>\n\n\n\n

      Faithfulness measures how faithful the model\u2019s answer is to the retrieved context. In other words, does the model really produce the answer based on the information given to it, or does it add something that is not in the context? This metric is important for detecting hallucination risk. Because even in a system that uses RAG, the model can sometimes receive the correct context but still produce an answer that is not fully faithful to the source. For this reason, it is not enough for an answer to be well-written. Sometimes, this is exactly the problem. The answer looks very smooth, very logical, and very convincing. But it has no real support in the source.<\/p>\n\n\n\n

      Faithfulness allows us to ask this question: Did this answer really come from the given context?<\/p>\n\n\n\n

      If the answer is not supported by the context, the system\u2019s access to information does not mean much on its own. Because the model may have moved away from the context during generation.<\/p>\n\n\n\n

      Answer Relevancy
      <\/h3>\n\n\n\n

      Answer Relevancy measures how suitable the answer is for the user\u2019s question. The system may say something correct, but it may not answer what the user actually asked.<\/p>\n\n\n\n

      This happens quite often in RAG systems. The model begins from a context that seems related and generates an answer that is not technically incorrect. Yet the answer drifts away from the core of the question.<\/p>\n\n\n\n

      Context Precision
      <\/h3>\n\n\n\n

      Context Precision is one of the metrics that measures the quality of the retrieval side. This metric checks whether the truly relevant context parts are placed in the upper ranks among the retrieved context parts. According to RAGAS documentation, Context Precision evaluates the retriever\u2019s ability to rank relevant chunks higher than irrelevant chunks.<\/p>\n\n\n\n

      Why is this important?<\/p>\n\n\n\n

      Because RAG systems usually give more than one context part to the model. If the most relevant information stays lower in the list, and weaker or unrelated information is at the top, the chance of the model producing the correct answer decreases. So, the system may have retrieved the correct information. But the correct information may be lost inside the crowd.<\/p>\n\n\n\n

      Context Recall
      <\/h3>\n\n\n\n

      Context Recall tries to understand how much of the needed information is included in the retrieved context.<\/p>\n\n\n\n

      Sometimes, a RAG system brings clean and relevant context parts but still misses some critical piece of information required for the answer. When this happens, context precision can look strong while recall stays low. In other words, the retrieved information may be high-quality yet incomplete.<\/p>\n\n\n\n

      This difference is important because missing context can cause the model to guess in order to complete the answer. When guessing starts, hallucination risk also increases.<\/p>\n\n\n\n

      Reading the RAGAS metrics together
      <\/h2>\n\n\n\n

      Looking at each RAGAS metric on its own is useful. But the real value comes from reading them side by side. Because every metric exposes a different kind of problem.<\/p>\n\n\n\n

        \n
      • If Faithfulness is low, the model may be moving outside the context.<\/li>\n\n\n\n
      • If Answer Relevancy is low, the answer may not meet the real purpose of the question.<\/li>\n\n\n\n
      • If Context Precision is low, the system may be ranking unrelated or weak context parts too high.<\/li>\n\n\n\n
      • If Context Recall is low, the system may not be retrieving all the information needed for the answer.<\/li>\n<\/ul>\n\n\n\n

        This is what makes RAGAS important for me. It makes the layer where the system struggles more visible. Is the problem on the retrieval side? Is it on the generation side? Is the context missing? Is the context unrelated? Is the model not staying faithful to the context? Is the answer moving away from the question? When we cannot make this distinction, improving the RAG system also becomes harder. Because we do not know what we need to fix.<\/p>\n\n\n\n

        Does RAGAS solve everything?
        <\/h2>\n\n\n\n

        Even though RAGAS and similar metrics are very valuable, they should not be seen as an absolute truth mechanism on their own. Actually, this is true for all other metrics too.<\/p>\n\n\n\n

        In the LLM-as-a-Judge approach, the model that makes the evaluation can also make mistakes, act inconsistently, or have bias. In the LLM-as-a-Judge literature, it is also emphasized that consistency, reducing bias, and standardized evaluation design are important for the reliability of these systems. Because of this, RAGAS metrics should not be seen as a structure that completely replaces human review. They should be seen as a control layer that makes the evaluation process more scalable.<\/p>\n\n\n\n

        My conclusion from this research is this:<\/p>\n\n\n\n

        In RAG systems, talking about architecture alone is no longer sufficient. Building RAG matters. Approaches such as GraphRAG and hybrid retrieval matter for making the context stronger. But after all of that, measuring whether the system truly works reliably remains its own separate topic.<\/p>\n\n\n\n

        RAG makes information accessible.<\/p>\n\n\n\n

        GraphRAG and hybrid retrieval try to strengthen the context.<\/p>\n\n\n\n

        RAGAS helps measure how reliable this system really is.<\/p>\n\n\n\n

        References<\/strong><\/p>\n\n\n\n

        Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. RAGAs: Automated Evaluation of Retrieval Augmented Generation, EACL 2024.<\/p>\n\n\n\n

        RAGAS Documentation, Component-Wise Evaluation Metrics.<\/p>\n\n\n\n

        RAGAS Documentation, Context Precision.<\/p>\n\n\n\n

        Microsoft Learn, Evaluation metrics for LLM-generated content.<\/p>\n\n\n\n

        Gu et al., A Survey on LLM-as-a-Judge, arXiv 2024, updated 2025.<\/p>\n","protected":false},"excerpt":{"rendered":"

        My previous articles about RAG, hybrid approaches, and GraphRAG are already published. In those articles, I mostly focused on how information is retrieved, how context is built, and how company knowledge can be transferred to AI systems in a more meaningful way. Today,…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[46,21,32,42,19,40,41],"class_list":["post-34","post","type-post","status-publish","format-standard","hentry","category-artificial-intelligence","tag-ai-evaluation","tag-artificial-intelligence","tag-enterprise-ai","tag-hallucination","tag-human-oversight","tag-large-language-models","tag-rag"],"_links":{"self":[{"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/posts\/34","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/comments?post=34"}],"version-history":[{"count":3,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/posts\/34\/revisions"}],"predecessor-version":[{"id":79,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/posts\/34\/revisions\/79"}],"wp:attachment":[{"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/media?parent=34"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/categories?post=34"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/regentis.ai\/blog\/wp-json\/wp\/v2\/tags?post=34"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}