mlsanityData to 14 Oct 2026

Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models

arXiv 2502.18817 ↗First posted 26 Feb 2025cs.CLcited by 8

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs). However, existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation. LLM-based judgment models provide the potential to produce high-quality judgments, but they are highly sensitive to evaluation prompts, leading to inconsistencies when judging the output of RAG models. This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models. Specifically, ConsJudge prompts LLMs to generate different judgments based on various combinations of judgment dimensions, utilize the judge-consistency to evaluate these judgments and select the accepted and rejected judgments for DPO training. Our experiments show that ConsJudge can effectively provide more accurate judgments for optimizing RAG models across various RAG models and datasets. Further analysis reveals that judgments generated by ConsJudge have a high agreement with the superior LLM. All codes are available at https://github.com/OpenBMB/ConsJudge.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 59 of its 76 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

59 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework2 Aug
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 56

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  12. 2024A Survey on LLM-as-a-Judgecited by 1,423
38 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  4. 2023Teaching Large Language Models to Self-Debugcited by 988
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  7. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  8. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  13. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  14. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  15. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  16. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  17. 2019ELI5: Long Form Question Answeringcited by 575
  18. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  19. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  20. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  21. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  22. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  23. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  24. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  25. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  26. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  27. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  28. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  29. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  30. 2021Efficient Nearest Neighbor Language Modelscited by 93
  31. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  32. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  33. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  34. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  35. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  36. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  37. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  38. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 23

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelinescited by 4
  2. 2025MHTS: Multi-Hop Tree Structure Framework for Generating Difficulty-Controllable QA Datasets for RAG Evaluationcited by 3
  3. 2025HetaRAG: Hybrid Deep Retrieval-Augmented Generation across Heterogeneous Data Storescited by 3
  4. 2025MAFA: A multi-agent framework for annotationcited by 2
  5. 2026CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Modelscited by 1
  6. 2026The Magnitude Mirage: Rethinking Confidence for Reasoning-Intensive Retrievalcited by 1
  7. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generationcited by 0
  8. 2026ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Lighter and Robust Evaluation for Retrieval Augmented Generation
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  3. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  4. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  5. 2025Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation
  6. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  7. 2026LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
  8. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems