# Underreporting of errors in NLG output, and what to do about it

arXiv: https://arxiv.org/abs/2108.01182 · HTML: https://mlsanity.com/paper/2108.01182/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 67 references that are in the map.
- First posted on arXiv 2 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [With Little Power Comes Great Responsibility](https://mlsanity.com/paper/2010.06595/) (2020)
- [Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge](https://mlsanity.com/paper/1901.07931/) (2019)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [Semantic Noise Matters for Neural Natural Language Generation](https://mlsanity.com/paper/1911.03905/) (2019)
- [A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems](https://mlsanity.com/paper/2011.03992/) (2020)
- [Findings of the E2E NLG Challenge](https://mlsanity.com/paper/1810.01170/) (2018)
- [Preregistering NLP Research](https://mlsanity.com/paper/2103.06944/) (2021)
- [Generating Intelligible Plumitifs Descriptions: Use Case Application with Ethical Considerations](https://mlsanity.com/paper/2011.12183/) (2020)
- [Room for improvement in automatic image description: an error analysis](https://mlsanity.com/paper/1704.04198/) (2017)

## Built on by (most cited first, 19 of 19)

- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications](https://mlsanity.com/paper/2205.06828/) (2022)
- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- [Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems](https://mlsanity.com/paper/2212.09180/) (2022)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [TabGenie: A Toolkit for Table-to-Text Generation](https://mlsanity.com/paper/2302.14169/) (2023)
- [Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo](https://mlsanity.com/paper/2404.04103/) (2024)
- [LLM-as-a-qualitative-judge: automating error analysis in natural language generation](https://mlsanity.com/paper/2506.09147/) (2025)
- [Implicit causality in GPT-2: a case study](https://mlsanity.com/paper/2212.04348/) (2022)
- [Natural Answer Generation: From Factoid Answer to Full-length Answer using Grammar Correction](https://mlsanity.com/paper/2112.03849/) (2021)
- [Comparing informativeness of an NLG chatbot vs graphical app in diet-information domain](https://mlsanity.com/paper/2206.13435/) (2022)
- [Guidance in Radiology Report Summarization: An Empirical Evaluation and Error Analysis](https://mlsanity.com/paper/2307.12803/) (2023)
- [Question Generation in Knowledge-Driven Dialog: Explainability and Evaluation](https://mlsanity.com/paper/2404.07836/) (2024)
- [Evaluating NLG systems: A brief introduction](https://mlsanity.com/paper/2303.16742/) (2023)
- [Probing Omissions and Distortions in Transformer-based RDF-to-Text Models](https://mlsanity.com/paper/2409.16707/) (2024)
- [Integration of LLM Quality Assurance into an NLG System](https://mlsanity.com/paper/2501.16078/) (2025)
- [Natural Language Generation](https://mlsanity.com/paper/2503.16728/) (2025)

## Abstract (arXiv)

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by `state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.