mlsanityData to 14 Oct 2026

Automatic Evaluation of Attribution by Large Language Models

arXiv 2305.06311 ↗First posted 10 May 2023cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

A recent focus of large language model (LLM) development, as exemplified by generative search engines, is to incorporate external references to generate and support its claims. However, evaluating the attribution, i.e., verifying whether the generated statement is fully supported by the cited reference, remains an open problem. Although human evaluation is common practice, it is costly and time-consuming. In this paper, we investigate the automatic evaluation of attribution given by LLMs. We begin by defining different types of attribution errors, and then explore two approaches for automatic evaluation: prompting LLMs and fine-tuning smaller LMs. The fine-tuning data is repurposed from related tasks such as question answering, fact-checking, natural language inference, and summarization. We manually curate a set of test examples covering 12 domains from a generative search engine, New Bing. Our results on this curated test set and simulated examples from existing benchmarks highlight both promising signals and challenges. We hope our problem formulation, testbeds, and findings will help lay the foundation for future studies on this important problem.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 62 of its 92 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

62 of 92 references.

References · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  13. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  14. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  16. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  17. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  18. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  19. 2023BloombergGPT: A Large Language Model for Financecited by 939
  20. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  21. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  22. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  23. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  24. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  25. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  26. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  27. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  28. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  29. 2023GPTScore: Evaluate as You Desirecited by 361
  30. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  31. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  32. 2023Tool Learning with Foundation Modelscited by 302
  33. 2022Teaching language models to support answers with verified quotescited by 292
  34. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  35. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  36. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  37. 2021Internet-Augmented Dialogue Generationcited by 217
  38. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 21; 2024: 24; 2025: 17; 2026: 10.

72 papers.

Sort:

By area:

  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  9. 2023Conformal Language Modelingcited by 170
  10. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  11. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  12. 2023Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensionscited by 112
38 more
  1. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  2. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  3. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  4. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  5. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  6. 2023CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-based Toolscited by 29
  7. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  8. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  9. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 27
  10. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  11. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  12. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 25
  13. 2023Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attributioncited by 24
  14. 2024FormalAlign: Automated Alignment Evaluation for Autoformalizationcited by 24
  15. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  16. 2024How well do LLMs cite relevant medical references? An evaluation framework and analysescited by 21
  17. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  18. 2024Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answeringcited by 20
  19. 2023Zero-Resource Hallucination Prevention for Large Language Modelscited by 19
  20. 2024Energy-based Automated Model Evaluationcited by 17
  21. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  22. 2024LitLLMs, LLMs for Literature Review: Are we there yet?cited by 14
  23. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  24. 2025Credible Plan-Driven RAG Method for Multi-Hop Question Answeringcited by 12
  25. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  26. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  27. 2024ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphscited by 11
  28. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  29. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  30. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  31. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applicationscited by 7
  32. 2025Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extractioncited by 7
  33. 2024DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learningcited by 5
  34. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  35. 2023X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphscited by 3
  36. 2024SciDaSynth: Interactive Structured Data Extraction from Scientific Literature with Large Language Modelcited by 3
  37. 2025The Attribution Crisis in LLM Search Resultscited by 3
  38. 2026ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agentscited by 3

50 most cited of 72 All 72 (JSON)