mlsanityData to 7 Oct 2026

Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA

arXiv 2603.24580 ↗First posted 25 Mar 2026cs.CL cs.AI cs.CYcited by 0

On the map: Retrieval-augmented generation → Legal and financial RAG

Abstract (arXiv)

Retrieval-augmented generation (RAG) systems are increasingly used to analyze complex policy documents, but achieving sufficient reliability for expert usage remains challenging in domains characterized by dense legal language and evolving, overlapping regulatory frameworks. We study the application of RAG to AI governance and policy analysis using the AI Governance and Regulatory Archive (AGORA) corpus, a curated collection of 947 AI policy documents. Our system combines a ColBERT-based retriever fine-tuned with contrastive learning and a generator aligned to human preferences using Direct Preference Optimization (DPO). We construct synthetic queries and collect pairwise preferences to adapt the system to the policy domain. Through experiments evaluating retrieval quality, answer relevance, and faithfulness, we find that domain-specific fine-tuning improves retrieval metrics but does not consistently improve end-to-end question answering performance. In some cases, stronger retrieval counterintuitively leads to more confident hallucinations when relevant documents are absent from the corpus. These results highlight a key concern for those building policy-focused RAG systems: improvements to individual components do not necessarily translate to more reliable answers. Our findings provide practical insights for designing grounded question-answering systems over dynamic regulatory corpora.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 41 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

22 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain19 Aug

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  8. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 115
  9. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  10. 2023Can GPT-3 Perform Statutory Reasoning?cited by 71
  11. 2019Question Answering for Privacy Policies: Combining Computational and Legal Perspectivescited by 65
  12. 2022GPT Takes the Bar Examcited by 65
9 more
  1. 2020PolicyQA: A Reading Comprehension Dataset for Privacy Policiescited by 31
  2. 2022Worldwide AI Ethics: a review of 200 guidelines and recommendations for AI governancecited by 28
  3. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  4. 2019U.S. Public Opinion on the Governance of Artificial Intelligencecited by 11
  5. 2024PA-RAG: RAG Alignment via Multi-Perspective Preference Optimizationcited by 11
  6. 2025Syntriever: How to Train Your Retriever with Synthetic Data from LLMscited by 11
  7. 2023BLT: Can Large Language Models Handle Basic Legal Text?cited by 10
  8. 2025On Synthetic Data Strategies for Domain-Specific Generative Retrievalcited by 5
  9. 2026Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Textcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PolicyBot - Reliable Question Answering over Policy Documents
  2. 2024HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications
  3. 2026Navigating Global AI Regulation: A Multi-Jurisdictional Retrieval-Augmented Generation System
  4. 2025Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation
  5. 2026Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answering
  6. 2026Bridging Static and Agentic RAG for Taiwanese Historical Question Answering
  7. 2026MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAG
  8. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG