# AMR Similarity Metrics from Principles

arXiv: https://arxiv.org/abs/2001.10929 · HTML: https://mlsanity.com/paper/2001.10929/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2020.

## Built on by (most cited first, 22 of 22)

- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://mlsanity.com/paper/2204.06508/) (2022)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- [Translate, then Parse! A strong baseline for Cross-Lingual AMR Parsing](https://mlsanity.com/paper/2106.04565/) (2021)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Incorporating Graph Information in Transformer-based AMR Parsing](https://mlsanity.com/paper/2306.13467/) (2023)
- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating](https://mlsanity.com/paper/2205.12176/) (2022)
- [MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection](https://mlsanity.com/paper/2405.19285/) (2024)
- [AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing](https://mlsanity.com/paper/2306.10786/) (2023)
- [Retrieval-Augmented Semantic Parsing: Improving Generalization with Lexical Knowledge](https://mlsanity.com/paper/2412.10207/) (2024)
- [Survey of Abstract Meaning Representation: Then, Now, Future](https://mlsanity.com/paper/2505.03229/) (2025)
- [Rematch: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity](https://mlsanity.com/paper/2404.02126/) (2024)
- [A Theorem-Proving-Based Evaluation of Neural Semantic Parsing](https://mlsanity.com/paper/2510.11225/) (2025)
- [SMARAGD: Learning SMatch for Accurate and Rapid Approximate Graph Distance](https://mlsanity.com/paper/2203.13226/) (2022)
- [Tracing the Genealogies of Ideas with Large Language Model Embeddings](https://mlsanity.com/paper/2402.01661/) (2024)
- [Evaluation of Finetuned LLMs in AMR Parsing](https://mlsanity.com/paper/2508.05028/) (2025)
- [SWAN: Semantic Watermarking with Abstract Meaning Representation](https://mlsanity.com/paper/2605.04305/) (2026)
- [Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization](https://mlsanity.com/paper/2607.01972/) (2026)
- [How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing](https://mlsanity.com/paper/2610.13236/) (2026)

## Abstract (arXiv)

Different metrics have been proposed to compare Abstract Meaning Representation (AMR) graphs. The canonical Smatch metric (Cai and Knight, 2013) aligns the variables of two graphs and assesses triple matches. The recent SemBleu metric (Song and Gildea, 2019) is based on the machine-translation metric Bleu (Papineni et al., 2002) and increases computational efficiency by ablating the variable-alignment. In this paper, i) we establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning representations like AMR; ii) we undertake a thorough analysis of Smatch and SemBleu where we show that the latter exhibits some undesirable properties. For example, it does not conform to the identity of indiscernibles rule and introduces biases that are hard to control; iii) we propose a novel metric S2match that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. We assess its suitability and show its advantages over Smatch and SemBleu.