mlsanityData to 13 Oct 2026

Underreporting of errors in NLG output, and what to do about it

arXiv 2108.01182 ↗First posted 2 Aug 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Builds on 17 of its 67 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

17 of 67 references.

References · 17

  1. 2018Model Cards for Model Reportingcited by 1,076
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 992
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 867
  4. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 334
  6. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  7. 2017Challenges in Data-to-Document Generationcited by 226
  8. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 166
  9. 2020With Little Power Comes Great Responsibilitycited by 140
  10. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  11. 2020Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Modelcited by 77
  12. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
5 more
  1. 2020A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systemscited by 34
  2. 2018Findings of the E2E NLG Challengecited by 28
  3. 2021Preregistering NLP Researchcited by 25
  4. 2020Generating Intelligible Plumitifs Descriptions: Use Case Application with Ethical Considerationscited by 5
  5. 2017Room for improvement in automatic image description: an error analysiscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 6; 2023: 4; 2024: 4; 2025: 3.

19 papers.

Sort:

  1. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 344
  2. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  3. 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
  4. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  5. 2022Neural Pipeline for Zero-Shot Data-to-Text Generationcited by 19
  6. 2022Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systemscited by 9
  7. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  8. 2023TabGenie: A Toolkit for Table-to-Text Generationcited by 7
  9. 2024Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTocited by 3
  10. 2025LLM-as-a-qualitative-judge: automating error analysis in natural language generationcited by 3
  11. 2022Implicit causality in GPT-2: a case studycited by 2
  12. 2021Natural Answer Generation: From Factoid Answer to Full-length Answer using Grammar Correctioncited by 1
7 more
  1. 2022Comparing informativeness of an NLG chatbot vs graphical app in diet-information domaincited by 1
  2. 2023Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysiscited by 1
  3. 2024Question Generation in Knowledge-Driven Dialog: Explainability and Evaluationcited by 1
  4. 2023Evaluating NLG systems: A brief introductioncited by 0
  5. 2024Probing Omissions and Distortions in Transformer-based RDF-to-Text Modelscited by 0
  6. 2025Integration of LLM Quality Assurance into an NLG Systemcited by 0
  7. 2025Natural Language Generationcited by 0