mlsanityData to 14 Oct 2026

The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

arXiv 2501.03200 ↗First posted 6 Jan 2025cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

32 of 39 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2024Measuring short-form factuality in large language modelscited by 373
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  6. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  7. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  8. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  9. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  10. 2024Long-form factuality in large language modelscited by 155
  11. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  12. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
20 more
  1. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  2. 2023Calibrated Language Models Must Hallucinatecited by 115
  3. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  4. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  5. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  6. 2024Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Modelscited by 86
  7. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  8. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  9. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  10. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  11. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  12. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  13. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
  14. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 20
  15. 2023LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisationcited by 16
  16. 2024FactAlign: Long-form Factuality Alignment of Large Language Modelscited by 12
  17. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  18. 2024Evaluating the Factuality of Zero-shot Summarizers Across Varied Domainscited by 10
  19. 2024CoverBench: A Challenging Benchmark for Complex Claim Verificationcited by 6
  20. 2024Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 22.

52 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  3. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54
  4. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  5. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 25
  6. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24
  7. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  8. 2025MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documentscited by 21
  9. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  10. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  11. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
  12. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
38 more
  1. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
  2. 2025Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systemscited by 12
  3. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 12
  4. 2025CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmarkcited by 10
  5. 2025A Neurosymbolic Approach to Natural Language Formalization and Verificationcited by 9
  6. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
  7. 2025Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluationcited by 8
  8. 2025FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMscited by 8
  9. 2025How Does Response Length Affect Long-Form Factualitycited by 6
  10. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  11. 2026MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucinationcited by 5
  12. 2025ConSens: Assessing context grounding in open-book question answeringcited by 4
  13. 2025Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretationscited by 4
  14. 2025StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluationcited by 4
  15. 2026Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Modelscited by 4
  16. 2026On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatilitycited by 4
  17. 2025Towards Large Reasoning Models for Agriculturecited by 3
  18. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  19. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  20. 2025Reward Models are Metrics in a Trench Coatcited by 2
  21. 2026Multimodal Fact-Level Attribution for Verifiable Reasoningcited by 2
  22. 2026A Geometric Analysis of Small-sized Language Model Hallucinationscited by 2
  23. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Modelscited by 2
  24. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  25. 2026Thomson: Continual Learning of Frontier Models for SovereignAIcited by 2
  26. 2025The illusion of a perfect metric: Why evaluating AI's words is harder than it lookscited by 1
  27. 2025SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluationcited by 1
  28. 2026TRACE: Tourism Recommendation with Accountable Citation Evidencecited by 1
  29. 2025Enhancing Reliability across Short and Long-Form QA via Reinforcement Learningcited by 0
  30. 2026What do Geometric Hallucination Detection Metrics Actually Measure?cited by 0
  31. 2026Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insightscited by 0
  32. 2026Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoringcited by 0
  33. 2026APEX: Automated Prompt Engineering eXpert with Dynamic Data Selectioncited by 0
  34. 2026ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidencecited by 0
  35. 2026Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generatorcited by 0
  36. 2026AI and Authenticity in Islamic Research: A Critical Evaluation of Generative AI Reliability, Hallucination, and Source Fidelity in Quranic, Hadith, and Fiqh Knowledgecited by 0
  37. 2026Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledgecited by 0
  38. 2026MAP: A Benchmark on Multimodal Accessibility Planning for Real World Placescited by 0

50 most cited of 52 All 52 (JSON)