mlsanityData to 14 Oct 2026

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

arXiv 2412.13746 ↗First posted 18 Dec 2024cs.CL cs.AI cs.IRcited by 23

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

also in 1 more
Abstract (arXiv)

Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned training.We release our benchmark and code publicly at https://huggingface.co/datasets/jinzhuoran/RAG-RewardBench/ for future work.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 59 of its 89 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

59 of 89 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systems9 Nov
  2. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAG18 Jan
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection17 Oct

Other references · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
38 more
  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  4. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  6. 2019ELI5: Long Form Question Answeringcited by 575
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  12. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  13. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  14. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  15. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  16. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  17. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  18. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  19. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  20. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  21. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  22. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  23. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  24. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  25. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  26. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  27. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  28. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  29. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  30. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  31. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  32. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  33. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  34. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  35. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  36. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  37. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  38. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  5. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 20
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  7. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  8. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  10. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  11. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  12. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
11 more
  1. 2026Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planningcited by 5
  2. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  3. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  4. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  5. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  6. 2025RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generationcited by 1
  7. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  8. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  9. 2026On the Role of Citations in Preference Datacited by 0
  10. 2026CRISP: Cultural Reward Modeling for Implicit Situated Proprietycited by 0
  11. 2026LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
  2. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervision
  3. 2024PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization
  4. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  5. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  6. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  7. 2025RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning
  8. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?