mlsanityData to 7 Oct 2026

LM vs LM: Detecting Factual Errors via Cross Examination

arXiv 2305.13281 ↗First posted 22 May 2023cs.CLcited by 155

Not on the map. Why?

Cited by 27 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability. A natural question is whether such factual errors can be detected automatically. Inspired by truth-seeking mechanisms in law, we propose a factuality evaluation framework for LMs that is based on cross-examination. Our key idea is that an incorrect claim is likely to result in inconsistency with other claims that the model generates. To discover such inconsistencies, we facilitate a multi-turn interaction between the LM that generated the claim and another LM (acting as an examiner) which introduces questions to discover inconsistencies. We empirically evaluate our method on factual claims made by multiple recent LMs on four benchmarks, finding that it outperforms existing methods and baselines, often by a large gap. Our results demonstrate the potential of using interacting LMs for capturing factual errors.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

59 of 79 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2018AI safety via debatecited by 384
  13. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  14. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  15. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  16. 2020Calibration of Pre-trained Transformerscited by 307
  17. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  18. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  19. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  20. 2021Unsolved Problems in ML Safetycited by 291
  21. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  22. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  23. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  24. 2020Selective Question Answering under Domain Shiftcited by 228
  25. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  26. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  27. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  28. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  29. 2018Dialogue Natural Language Inferencecited by 135
  30. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  31. 2022Training Language Models with Memory Augmentationcited by 112
  32. 2018The Fact Extraction and VERification (FEVER) Shared Taskcited by 108
  33. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  34. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  35. 2022PEER: A Collaborative Language Modelcited by 86
  36. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  37. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  38. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 56; 2025: 54; 2026: 19.

155 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  4. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  5. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  6. 2023Challenges and Applications of Large Language Modelscited by 251
  7. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  8. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  9. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  10. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  11. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  12. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
38 more
  1. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  2. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  3. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  4. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  5. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  6. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  7. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  8. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  9. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  10. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  11. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  12. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  13. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  14. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  15. 2024Building Guardrails for Large Language Modelscited by 61
  16. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  17. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  18. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  19. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  20. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  21. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  22. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  23. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  24. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  25. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  26. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 44
  27. 2023A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detectioncited by 42
  28. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  29. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  30. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  31. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  32. 2023Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectificationcited by 38
  33. 2024Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinationscited by 38
  34. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  35. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  36. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  37. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  38. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applicationscited by 32

50 most cited of 155 All 155 (JSON)