mlsanityData to 7 Oct 2026

A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating

arXiv 2205.12176 ↗First posted 24 May 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Evaluating the quality of generated text is difficult, since traditional NLG evaluation metrics, focusing more on surface form than meaning, often fail to assign appropriate scores. This is especially problematic for AMR-to-text evaluation, given the abstract nature of AMR. Our work aims to support the development and improvement of NLG evaluation metrics that focus on meaning, by developing a dynamic CheckList for NLG metrics that is interpreted by being organized around meaning-relevant linguistic phenomena. Each test instance consists of a pair of sentences with their AMR graphs and a human-produced textual semantic similarity or relatedness score. Our CheckList facilitates comparative evaluation of metrics and reveals strengths and weaknesses of novel and traditional metrics. We demonstrate the usefulness of CheckList by designing a new metric GraCo that computes lexical cohesion graphs over AMR concepts. Our analysis suggests that GraCo presents an interesting NLG metric worth future investigation and that meaning-oriented NLG metrics can profit from graph-based metric components using AMR.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 16 of its 48 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

16 of 48 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  7. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  10. 2020Evolution of Semantic Similarity -- A Surveycited by 83
  11. 2017Neural AMR: Sequence-to-Sequence Models for Parsing and Generationcited by 56
  12. 2018A Graph-to-Sequence Model for AMR-to-Text Generationcited by 56
4 more
  1. 2020Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMRcited by 25
  2. 2020AMR Similarity Metrics from Principlescited by 21
  3. 2021Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similaritycited by 15
  4. 2020A Human Evaluation of AMR-to-English Generation Systemscited by 13

Built on by

Cited by year

Citing papers per year: 2022: 5.

5 papers.

  1. 2022SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Featurescited by 33
  2. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10
  3. 2022Better Smatch = Better Parser? AMR evaluation is not so simple anymorecited by 9
  4. 2022DialGuide: Aligning Dialogue Model Behavior with Developer Guidelinescited by 3
  5. 2022Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representationcited by 2