mlsanityData to 14 Oct 2026

Measuring Attribution in Natural Language Generation Models

arXiv 2112.12870 ↗First posted 23 Dec 2021cs.CLcited by 256

Not on the map. Why?

Cited by 40 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world. In this work, we present a new evaluation framework entitled Attributable to Identified Sources (AIS) for assessing the output of natural language generation models, when such output pertains to the external world. We first define AIS and introduce a two-stage annotation pipeline for allowing annotators to appropriately evaluate model output according to AIS guidelines. We empirically validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset) via human evaluation studies that suggest that AIS could serve as a common framework for measuring whether model-generated statements are supported by underlying sources. We release guidelines for the human evaluation studies.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 23 Dec 2021.

Builds on

28 of 41 references.

References · 28

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  5. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  8. 2018QuAC : Question Answering in Contextcited by 381
  9. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  11. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  12. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
16 more
  1. 2017Challenges in Data-to-Document Generationcited by 226
  2. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  3. 2020Extractive Summarization as Text Matchingcited by 189
  4. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  5. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 148
  6. 2018Dialogue Natural Language Inferencecited by 135
  7. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  8. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  9. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  10. 2018Automated Fact Checking: Task formulations, methods and future directionscited by 96
  11. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 77
  12. 2021Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Featurescited by 74
  13. 2021DialFact: A Benchmark for Fact-Checking in Dialoguecited by 62
  14. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  15. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 51
  16. 2019The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agentscited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 64; 2024: 61; 2025: 42; 2026: 75.

256 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  6. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  7. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  8. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  10. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  11. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  12. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
38 more
  1. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  3. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  4. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  5. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  6. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 138
  7. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  9. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 112
  10. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  11. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 109
  12. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  13. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  14. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 96
  15. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  16. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 91
  17. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  18. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 72
  19. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  20. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 72
  21. 2023A Survey of Large Language Models Attributioncited by 71
  22. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  23. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 65
  24. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  25. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 57
  26. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 56
  27. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  28. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54
  29. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  30. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  31. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  32. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  33. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  34. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 50
  35. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  36. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  37. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  38. 2023DelucionQA: Detecting Hallucinations in Domain-specific Question Answeringcited by 43

50 most cited of 256 All 256 (JSON)