# Weisfeiler-Leman in the BAMBOO: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity

arXiv: https://arxiv.org/abs/2108.11949 · HTML: https://mlsanity.com/paper/2108.11949/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Wasserstein Weisfeiler-Lehman Graph Kernels](https://mlsanity.com/paper/1906.01277/) (2019)
- [Leveraging Abstract Meaning Representation for Knowledge Base Question Answering](https://mlsanity.com/paper/2012.01707/) (2020)
- [AMR Parsing as Graph Prediction with Latent Alignment](https://mlsanity.com/paper/1805.05286/) (2018)
- [Improving AMR Parsing with Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2010.01771/) (2020)
- [Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning](https://mlsanity.com/paper/1905.13370/) (2019)
- [Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR](https://mlsanity.com/paper/2008.08896/) (2020)
- [Core Semantic First: A Top-down Approach for AMR Parsing](https://mlsanity.com/paper/1909.04303/) (2019)
- [AMR Similarity Metrics from Principles](https://mlsanity.com/paper/2001.10929/) (2020)
- [Translate, then Parse! A strong baseline for Cross-Lingual AMR Parsing](https://mlsanity.com/paper/2106.04565/) (2021)
- [SemBleu: A Robust Metric for AMR Parsing Evaluation](https://mlsanity.com/paper/1905.10726/) (2019)
- [Evaluating Scoped Meaning Representations](https://mlsanity.com/paper/1802.08599/) (2018)
- [Bootstrapping Multilingual AMR with Contextual Word Alignments](https://mlsanity.com/paper/2102.02189/) (2021)
- [AMR Quality Rating with a Lightweight CNN](https://mlsanity.com/paper/2005.12187/) (2020)
- [Joint Learning of Sentence Embeddings for Relevance and Entailment](https://mlsanity.com/paper/1605.04655/) (2016)
- [Sentence Pair Scoring: Towards Unified Framework for Text Comprehension](https://mlsanity.com/paper/1603.06127/) (2016)
- [An overview of SPSA: recent development and applications](https://mlsanity.com/paper/2012.06952/) (2020)

## Built on by (most cited first, 16 of 16)

- [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://mlsanity.com/paper/2204.06508/) (2022)
- [SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features](https://mlsanity.com/paper/2206.07023/) (2022)
- [SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://mlsanity.com/paper/2305.06993/) (2023)
- ["You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation](https://mlsanity.com/paper/2310.17793/) (2023)
- [FastKASSIM: A Fast Tree Kernel-Based Syntactic Similarity Metric](https://mlsanity.com/paper/2203.08299/) (2022)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Incorporating Graph Information in Transformer-based AMR Parsing](https://mlsanity.com/paper/2306.13467/) (2023)
- [A Dynamic, Interpreted CheckList for Meaning-oriented NLG Metric Evaluation -- through the Lens of Semantic Similarity Rating](https://mlsanity.com/paper/2205.12176/) (2022)
- [AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing](https://mlsanity.com/paper/2306.10786/) (2023)
- [Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://mlsanity.com/paper/2210.03018/) (2022)
- [Survey of Abstract Meaning Representation: Then, Now, Future](https://mlsanity.com/paper/2505.03229/) (2025)
- [Rematch: Robust and Efficient Matching of Local Knowledge Graphs to Improve Structural and Semantic Similarity](https://mlsanity.com/paper/2404.02126/) (2024)
- [SMARAGD: Learning SMatch for Accurate and Rapid Approximate Graph Distance](https://mlsanity.com/paper/2203.13226/) (2022)
- [AMR4NLI: Interpretable and robust NLI measures from semantic graphs](https://mlsanity.com/paper/2306.00936/) (2023)
- [Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization](https://mlsanity.com/paper/2607.01972/) (2026)
- [How Reliable are AMR Parser Outputs? Uncertainty Estimation for AMR Parsing](https://mlsanity.com/paper/2610.13236/) (2026)

## Abstract (arXiv)

Several metrics have been proposed for assessing the similarity of (abstract) meaning representations (AMRs), but little is known about how they relate to human similarity ratings. Moreover, the current metrics have complementary strengths and weaknesses: some emphasize speed, while others make the alignment of graph structures explicit, at the price of a costly alignment step. In this work we propose new Weisfeiler-Leman AMR similarity metrics that unify the strengths of previous metrics, while mitigating their weaknesses. Specifically, our new metrics are able to match contextualized substructures and induce n:m alignments between their nodes. Furthermore, we introduce a Benchmark for AMR Metrics based on Overt Objectives (BAMBOO), the first benchmark to support empirical assessment of graph-based MR similarity metrics. BAMBOO maximizes the interpretability of results by defining multiple overt objectives that range from sentence similarity objectives to stress tests that probe a metric's robustness against meaning-altering and meaning-preserving graph transformations. We show the benefits of BAMBOO by profiling previous metrics and our own metrics. Results indicate that our novel metrics may serve as a strong baseline for future work.