# Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

arXiv: https://arxiv.org/abs/2008.08896 · HTML: https://mlsanity.com/paper/2008.08896/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2020.

## Built on by (most cited first, 25 of 25)

- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains](https://mlsanity.com/paper/2402.00559/) (2024)
- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation](https://mlsanity.com/paper/2104.07555/) (2021)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2108.02859/) (2021)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating](https://mlsanity.com/paper/2205.12176/) (2022)
- [Faithful Chart Summarization with ChaTS-Pi](https://mlsanity.com/paper/2405.19094/) (2024)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics](https://mlsanity.com/paper/2412.16120/) (2024)
- [Contextualized Semantic Distance between Highly Overlapped Texts](https://mlsanity.com/paper/2110.01176/) (2021)
- [Graph-to-Text Generation with Dynamic Structure Pruning](https://mlsanity.com/paper/2209.07258/) (2022)
- [Investigating the Effect of Relative Positional Embeddings on AMR-to-Text Generation with Structural Adapters](https://mlsanity.com/paper/2302.05900/) (2023)
- [Unsupervised Full Constituency Parsing with Neighboring Distribution Divergence](https://mlsanity.com/paper/2110.15931/) (2021)
- [SMARAGD: Learning SMatch for Accurate and Rapid Approximate Graph Distance](https://mlsanity.com/paper/2203.13226/) (2022)
- [A Survey : Neural Networks for AMR-to-Text](https://mlsanity.com/paper/2206.07328/) (2022)
- [Counterfactual Edits for Generative Evaluation](https://mlsanity.com/paper/2303.01555/) (2023)
- [AMR4NLI: Interpretable and robust NLI measures from semantic graphs](https://mlsanity.com/paper/2306.00936/) (2023)

## Abstract (arXiv)

Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing MFβ, a decomposable metric that builds on two pillars. The first is the principle of meaning preservation M: it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (fine-grained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form F that measures the linguistic quality of the generated text, which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores. Since MFβ does not necessarily rely on gold AMRs, it may extend to other text generation tasks.