mlsanityData to 7 Oct 2026

RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge

arXiv 2311.08147 ↗First posted 14 Nov 2023cs.CL cs.AIcited by 51

Not on the map. Why?

Cited by 34 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

LLMs and AI chatbots have improved people's efficiency in various fields. However, the necessary knowledge for answering the question may be beyond the models' knowledge boundaries. To mitigate this issue, many researchers try to introduce external knowledge, such as knowledge graphs and Internet contents, into LLMs for up-to-date information. However, the external information from the Internet may include counterfactual information that will confuse the model and lead to an incorrect response. Thus there is a pressing need for LLMs to possess the ability to distinguish reliable information from external knowledge. Therefore, to evaluate the ability of LLMs to discern the reliability of external knowledge, we create a benchmark from existing knowledge bases. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information. Evaluation results show that existing LLMs are susceptible to interference from unreliable external knowledge with counterfactual information, and simple intervention methods make limited contributions to the alleviation of this issue.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

24 of 38 references.

References · 24

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  10. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
12 more
  1. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  2. 2023Tool Learning with Foundation Modelscited by 302
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  4. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  5. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  6. 2022RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Groundingcited by 47
  7. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 44
  8. 2018EventKG: A Multilingual Event-Centric Temporal Knowledge Graphcited by 43
  9. 2023Retrieving Supporting Evidence for LLMs Generated Answerscited by 21
  10. 2021Uncertainty Estimation and Out-of-Distribution Detection for Counterfactual Explanations: Pitfalls and Solutionscited by 16
  11. 2021Understanding Jargon: Combining Extraction and Generation for Definition Modelingcited by 15
  12. 2020SemEval-2020 Task 5: Counterfactual Recognitioncited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 23; 2025: 18; 2026: 9.

51 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  4. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  5. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  6. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  7. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  8. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  9. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 51
  10. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  11. 2025SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Modelcited by 31
  12. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
38 more
  1. 2024HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Modelscited by 27
  2. 2024Characterizing LLM Abstention Behavior in Science QA with Context Perturbationscited by 26
  3. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  4. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  5. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  6. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  7. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  8. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversitycited by 11
  9. 2025MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluationcited by 11
  10. 2025Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Datacited by 10
  11. 2025QE-RAG: A Robust Retrieval-Augmented Generation Benchmark for Query Entry Errorscited by 10
  12. 2024Evaluating the External and Parametric Knowledge Fusion of Large Language Modelscited by 9
  13. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  14. 2024Enhancing Robustness in Large Language Models: Prompting for Mitigating the Impact of Irrelevant Informationcited by 8
  15. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systemscited by 8
  16. 2025Investigating the Robustness of Deductive Reasoning with Large Language Modelscited by 8
  17. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  18. 2024Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generationcited by 6
  19. 2024Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALMscited by 5
  20. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Surveycited by 5
  21. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Modelscited by 5
  22. 2024VERA: Validation and Evaluation of Retrieval-Augmented Systemscited by 4
  23. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
  24. 2025ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systemscited by 4
  25. 2024InspectorRAGet: An Introspection Platform for RAG Evaluationcited by 3
  26. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  27. 2025Can we Evaluate RAGs with Synthetic Data?cited by 2
  28. 2024Unlocking Multi-View Insights in Knowledge-Dense Retrieval-Augmented Generationcited by 1
  29. 2025Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systemscited by 1
  30. 2025RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAGcited by 1
  31. 2026Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Frameworkcited by 1
  32. 2025Balancing Knowledge Updates: Toward Unified Modular Editing in LLMscited by 0
  33. 2025Quantifying Document Impact in RAG-LLMscited by 0
  34. 2026Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agentscited by 0
  35. 2026MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studiescited by 0
  36. 2026FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturingcited by 0
  37. 2026TrustRAG: Blockchain-Enhanced RAG via Committee-Based Credibility Scoringcited by 0
  38. 2026MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Modelscited by 0

50 most cited of 51 All 51 (JSON)