# SemBleu: A Robust Metric for AMR Parsing Evaluation

arXiv: https://arxiv.org/abs/1905.10726 · HTML: https://mlsanity.com/paper/1905.10726/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2019.

## Built on by (most cited first, 12 of 12)

- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- [Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity](https://mlsanity.com/paper/2108.11949/) (2021)
- ["You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation](https://mlsanity.com/paper/2310.17793/) (2023)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Visual Semantic Parsing: From Images to Abstract Meaning Representation](https://mlsanity.com/paper/2210.14862/) (2022)
- [Survey of Abstract Meaning Representation: Then, Now, Future](https://mlsanity.com/paper/2505.03229/) (2025)
- [A Transition-based Parser for Unscoped Episodic Logical Forms](https://mlsanity.com/paper/2103.08759/) (2021)
- [Learning from Multiple Sources for Data-to-Text and Text-to-Data](https://mlsanity.com/paper/2302.11269/) (2023)
- [Rematch: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity](https://mlsanity.com/paper/2404.02126/) (2024)
- [SMARAGD: Learning SMatch for Accurate and Rapid Approximate Graph Distance](https://mlsanity.com/paper/2203.13226/) (2022)
- [Evaluation of Finetuned LLMs in AMR Parsing](https://mlsanity.com/paper/2508.05028/) (2025)
- [How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing](https://mlsanity.com/paper/2610.13236/) (2026)

## Abstract (arXiv)

Evaluating AMR parsing accuracy involves comparing pairs of AMR graphs. The major evaluation metric, SMATCH (Cai and Knight, 2013), searches for one-to-one mappings between the nodes of two AMRs with a greedy hill-climbing algorithm, which leads to search errors. We propose SEMBLEU, a robust metric that extends BLEU (Papineni et al., 2002) to AMRs. It does not suffer from search errors and considers non-local correspondences in addition to local ones. SEMBLEU is fully content-driven and punishes situations where a system's output does not preserve most information from the input. Preliminary experiments on both sentence and corpus levels show that SEMBLEU has slightly higher consistency with human judgments than SMATCH. Our code is available at http://github.com/freesunshine0316/sembleu.