mlsanityData to 7 Oct 2026

Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks

arXiv 2602.06526 ↗First posted 6 Feb 2026cs.CL cs.AIcited by 1

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Information retrieval (IR) evaluation remains challenging due to incomplete IR benchmark datasets that contain unlabeled relevant chunks. While LLMs and LLM-human hybrid strategies reduce costly human effort, they remain prone to LLM overconfidence and ineffective AI-to-human escalation. To address this, we propose DREAM, a multi-round debate-based relevance assessment framework with LLM agents, built on opposing initial stances and iterative reciprocal critique. Through our agreement-based debate, it yields more accurate labeling for certain cases and more reliable AI-to-human escalation for uncertain ones, achieving 95.2% labeling accuracy with only 3.5% human involvement. Using DREAM, we build BRIDGE, a refined benchmark that mitigates evaluation bias and enables fairer retriever comparison by uncovering 29,824 missing relevant chunks. We then re-benchmark IR systems and extend evaluation to RAG, showing that unaddressed holes not only distort retriever rankings but also drive retrieval-generation misalignment. The relevance assessment framework is available at https: //github.com/DISL-Lab/DREAM-ICLR-26; and the BRIDGE dataset is available at https://github.com/DISL-Lab/BRIDGE-Benchmark.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 70 of its 79 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

70 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation31 Dec

Other references · 69

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
38 more
  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  3. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  4. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  6. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  7. 2020Overview of the TREC 2019 deep learning trackcited by 282
  8. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  9. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  10. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  11. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  12. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
  13. 2023Large language models can accurately predict searcher preferencescited by 129
  14. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  15. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  16. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  17. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  18. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agentcited by 92
  19. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  20. 2023Context-faithful Prompting for Large Language Modelscited by 76
  21. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  22. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  23. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  24. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 57
  25. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  26. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactionscited by 50
  27. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  28. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  29. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 41
  30. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 39
  31. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  32. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Modelscited by 34
  33. 2023One-Shot Labeling for Automatic Relevance Estimationcited by 31
  34. 2025Large Language Models are overconfident and amplify human biascited by 30
  35. 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
  36. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  37. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 17
  38. 2024Exploring the Best Practices of Query Expansion with Large Language Modelscited by 16

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Makes a Sale? Simulating End-to-End Seller--Buyer Retail Dynamics with LLM Agentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval
  2. 2026HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems
  3. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe
  4. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  5. 2026Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration
  6. 2025A Multi-Agent Perspective on Modern Information Retrieval
  7. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking
  8. 2026SAGE: Benchmarking and Improving Retrieval for Deep Research Agents