# SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

arXiv: https://arxiv.org/abs/2305.06993 · HTML: https://mlsanity.com/paper/2305.06993/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 23 of 23)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Wasserstein Weisfeiler-Lehman Graph Kernels](https://mlsanity.com/paper/1906.01277/) (2019)
- [AMR Parsing as Sequence-to-Graph Transduction](https://mlsanity.com/paper/1905.08704/) (2019)
- [Structural Adapters in Pretrained Language Models for AMR-to-text Generation](https://mlsanity.com/paper/2103.09120/) (2021)
- [AMR Parsing via Graph-Sequence Iterative Inference](https://mlsanity.com/paper/2004.05572/) (2020)
- [AMR Parsing as Graph Prediction with Latent Alignment](https://mlsanity.com/paper/1805.05286/) (2018)
- [An Incremental Parser for Abstract Meaning Representation](https://mlsanity.com/paper/1608.06111/) (2016)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR](https://mlsanity.com/paper/2008.08896/) (2020)
- [Core Semantic First: A Top-down Approach for AMR Parsing](https://mlsanity.com/paper/1909.04303/) (2019)
- [AMR Similarity Metrics from Principles](https://mlsanity.com/paper/2001.10929/) (2020)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- [Translate, then Parse! A strong baseline for Cross-Lingual AMR Parsing](https://mlsanity.com/paper/2106.04565/) (2021)
- [Ensembling Graph Predictions for AMR Parsing](https://mlsanity.com/paper/2110.09131/) (2021)
- [Promoting Graph Awareness in Linearized Graph-to-Text Generation](https://mlsanity.com/paper/2012.15793/) (2020)
- [SemBleu: A Robust Metric for AMR Parsing Evaluation](https://mlsanity.com/paper/1905.10726/) (2019)
- [Evaluating Scoped Meaning Representations](https://mlsanity.com/paper/1802.08599/) (2018)
- [Fast semantic parsing with well-typedness guarantees](https://mlsanity.com/paper/2009.07365/) (2020)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Universal Decompositional Semantic Parsing](https://mlsanity.com/paper/1910.10138/) (2019)
- [AMR Normalization for Fairer Evaluation](https://mlsanity.com/paper/1909.01568/) (2019)
- [Neural-Symbolic Inference for Robust Autoregressive Graph Parsing via Compositional Uncertainty Quantification](https://mlsanity.com/paper/2301.11459/) (2023)

## Built on by (most cited first, 21 of 21)

- ["You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation](https://mlsanity.com/paper/2310.17793/) (2023)
- [Interpretable Text Embeddings and Text Similarity Explanation: A Survey](https://mlsanity.com/paper/2502.14862/) (2025)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions](https://mlsanity.com/paper/2406.04286/) (2024)
- [AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite](https://mlsanity.com/paper/2312.03480/) (2023)
- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection](https://mlsanity.com/paper/2405.19285/) (2024)
- [Assessing the Sensitivity and Alignment of FOL Closeness Metrics](https://mlsanity.com/paper/2501.08613/) (2025)
- [Scope-enhanced Compositional Semantic Parsing for DRT](https://mlsanity.com/paper/2407.01899/) (2024)
- [Survey of Abstract Meaning Representation: Then, Now, Future](https://mlsanity.com/paper/2505.03229/) (2025)
- [When Does Meaning Backfire? Investigating the Role of AMRs in NLI](https://mlsanity.com/paper/2506.14613/) (2025)
- [A Theorem-Proving-Based Evaluation of Neural Semantic Parsing](https://mlsanity.com/paper/2510.11225/) (2025)
- [AMR4NLI: Interpretable and robust NLI measures from semantic graphs](https://mlsanity.com/paper/2306.00936/) (2023)
- [HuAMR: A Hungarian AMR Parser and Dataset](https://mlsanity.com/paper/2502.20552/) (2025)
- [Is neural semantic parsing good at ellipsis resolution, or isn't it?](https://mlsanity.com/paper/2508.00121/) (2025)
- [Evaluation of Finetuned LLMs in AMR Parsing](https://mlsanity.com/paper/2508.05028/) (2025)
- [SETUP: Sentence-level English-To-Uniform Meaning Representation Parser](https://mlsanity.com/paper/2512.07068/) (2025)
- [Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization](https://mlsanity.com/paper/2607.01972/) (2026)
- [Quizzing the Translation: A Prover-Grounded Evaluation Metric for NLrightarrowFOL](https://mlsanity.com/paper/2609.33612/) (2026)
- [How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing](https://mlsanity.com/paper/2610.13236/) (2026)
- [Structural Automata for Neural Semantic Parsing](https://mlsanity.com/paper/2610.13896/) (2026)

## Abstract (arXiv)

The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp