# Generation Challenges: Results of the Accuracy Evaluation Shared Task

arXiv: https://arxiv.org/abs/2108.05644 · HTML: https://mlsanity.com/paper/2108.05644/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 9 of 9)

- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Data-to-text Generation with Entity Modeling](https://mlsanity.com/paper/1906.03221/) (2019)
- [A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems](https://mlsanity.com/paper/2011.03992/) (2020)
- [Revisiting Challenges in Data-to-Text Generation with Fact Grounding](https://mlsanity.com/paper/2001.03830/) (2020)
- [A Hierarchical Model for Data-to-Text Generation](https://mlsanity.com/paper/1912.10011/) (2019)
- [Table-to-Text Generation with Effective Hierarchical Encoder on Three Dimensions (Row, Column and Time)](https://mlsanity.com/paper/1909.02304/) (2019)
- [Learning to Select, Track, and Generate for Data-to-Text](https://mlsanity.com/paper/1907.09699/) (2019)
- [Scrutinizer: A Mixed-Initiative Approach to Large-Scale, Data-Driven Claim Verification](https://mlsanity.com/paper/2003.06708/) (2020)

## Built on by (most cited first, 9 of 9)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [Data-to-text Generation with Variational Sequential Planning](https://mlsanity.com/paper/2202.13756/) (2022)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topics](https://mlsanity.com/paper/2403.08904/) (2024)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [A Data-Centric Approach To Generate Faithful and High Quality Patient Summaries with Large Language Models](https://mlsanity.com/paper/2402.15422/) (2024)
- [Probing Omissions and Distortions in Transformer-based RDF-to-Text Models](https://mlsanity.com/paper/2409.16707/) (2024)
- [Chatbots Output Meaningful (but Problematic) Language](https://mlsanity.com/paper/2606.02973/) (2026)

## Abstract (arXiv)

The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this task, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).