# A Human Evaluation of AMR-to-English Generation Systems

arXiv: https://arxiv.org/abs/2004.06814 · HTML: https://mlsanity.com/paper/2004.06814/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2020.

## Built on by (most cited first, 13 of 13)

- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [ToxCCIn: Toxic Content Classification with Interpretability](https://mlsanity.com/paper/2103.01328/) (2021)
- [Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization](https://mlsanity.com/paper/2212.10397/) (2022)
- [Biomedical Data-to-Text Generation via Fine-Tuning Transformers](https://mlsanity.com/paper/2109.01518/) (2021)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating](https://mlsanity.com/paper/2205.12176/) (2022)
- [Sentence Smith: Controllable Edits for Evaluating Text Embeddings](https://mlsanity.com/paper/2502.14734/) (2025)
- [Spanish Abstract Meaning Representation: Annotation of a General Corpus](https://mlsanity.com/paper/2204.07663/) (2022)
- [Generating Text from Uniform Meaning Representation](https://mlsanity.com/paper/2502.11973/) (2025)
- [A Survey : Neural Networks for AMR-to-Text](https://mlsanity.com/paper/2206.07328/) (2022)
- [Abstract Meaning Representation for Hospital Discharge Summarization](https://mlsanity.com/paper/2506.14101/) (2025)
- [Graph Alignment Topology as an Inductive Bias for Grounding Detection](https://mlsanity.com/paper/2605.22963/) (2026)
- [Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation](https://mlsanity.com/paper/2609.13581/) (2026)

## Abstract (arXiv)

Most current state-of-the art systems for generating English text from Abstract Meaning Representation (AMR) have been evaluated only using automated metrics, such as BLEU, which are known to be problematic for natural language generation. In this work, we present the results of a new human evaluation which collects fluency and adequacy scores, as well as categorization of error types, for several recent AMR generation systems. We discuss the relative quality of these systems and how our results compare to those of automatic metrics, finding that while the metrics are mostly successful in ranking systems overall, collecting human judgments allows for more nuanced comparisons. We also analyze common errors made by these systems.