mlsanityData to 14 Oct 2026

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

arXiv 2609.39229 ↗First posted 30 Sep 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate whether a panel of cheap open-weight judges (4--9B) can be aggregated to stand in for a frontier one, what the substitution sacrifices, and when it is worth making. We propose RAIM, an aggregation scheme robust to the members' correlated errors, coupling a cross-fitted stacked logistic regression with an admissibility test that, read from the members' own outputs, identifies when aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM with ten judges from disjoint families across eight faithfulness benchmarks. Against Claude Sonnet, the panel retains a median 93% of its Cohen's kappa and gives up only 2.9 points of balanced accuracy on average; read as paired differences, it clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving four unresolved. At a sixty-fourth of the frontier's inference price, the operative expense is a one-time in-domain calibration on 50--100 labelled records. The panel is also competitive with purpose-trained detectors on their home benchmarks (within 1.3 accuracy points of GPT-4o and 1.9 of the LLM-AggreFact leader), and beats the strongest one we reran by 6 points on our grounded sets. Whether aggregation pays depends on the members themselves: where several capable members err on different items, the panel improves on its best judge and approaches the frontier; where one dominates, the stacker recovers the leader, and only there does the frontier remain materially ahead. Both conditions are read off the calibration set at no further cost, so a cheap panel can stand in for a frontier one wherever this audit admits it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 61 of its 91 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

61 of 91 references.

References · 61

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  9. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  11. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  12. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
38 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  5. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  6. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  7. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  8. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  9. 2023Generative Judge for Evaluating Alignmentcited by 169
  10. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  11. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  12. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  13. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  14. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  15. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  16. 2025Great Models Think Alike and this Undermines AI Oversightcited by 58
  17. 2025Correlated Errors in Large Language Modelscited by 45
  18. 2025MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Modelscited by 34
  19. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applicationscited by 34
  20. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 26
  21. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
  22. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 25
  23. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  24. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  25. 2015Unsupervised Ensemble Learning with Dependent Classifierscited by 19
  26. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  27. 2024Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Taskscited by 18
  28. 2025A comprehensive taxonomy of hallucinations in Large Language Modelscited by 18
  29. 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Datacited by 14
  30. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
  31. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  32. 2025Tuning LLM Judge Design Decisions for 1/1000 of the Costcited by 11
  33. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  34. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metricscited by 8
  35. 2024SkillAggregation: Reference-free LLM-Dependent Aggregationcited by 6
  36. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 5
  37. 2026Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Liftcited by 5
  38. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.