# PMB5: Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks

arXiv: https://arxiv.org/abs/2404.08354 · HTML: https://mlsanity.com/paper/2404.08354/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 60 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations](https://mlsanity.com/paper/1702.03964/) (2017)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [Exploring Neural Methods for Parsing Discourse Representation Structures](https://mlsanity.com/paper/1810.12579/) (2018)
- [Towards Universal Semantic Tagging](https://mlsanity.com/paper/1709.10381/) (2017)
- [Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation](https://mlsanity.com/paper/2306.00124/) (2023)
- [Better Smatch = Better Parser? AMR evaluation is not so simple anymore](https://mlsanity.com/paper/2210.06461/) (2022)
- [Semantic Graph Parsing with Recurrent Neural Network DAG Grammars](https://mlsanity.com/paper/1910.00051/) (2019)
- [AMR Parsing is Far from Solved: GrAPES, the Granular AMR Parsing Evaluation Suite](https://mlsanity.com/paper/2312.03480/) (2023)
- [The First Shared Task on Discourse Representation Structure Parsing](https://mlsanity.com/paper/2005.13399/) (2020)
- [Semantic Tagging with Deep Residual Networks](https://mlsanity.com/paper/1609.07053/) (2016)
- [DRTS Parsing with Structure-Aware Encoding and Decoding](https://mlsanity.com/paper/2005.06901/) (2020)
- [What can we learn from Semantic Tagging?](https://mlsanity.com/paper/1808.09716/) (2018)
- [The Parallel Meaning Bank: A Framework for Semantically Annotating Multiple Languages](https://mlsanity.com/paper/2012.14854/) (2020)

## Built on by (most cited first, 5 of 5)

- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [Scope-enhanced Compositional Semantic Parsing for DRT](https://mlsanity.com/paper/2407.01899/) (2024)
- [Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions](https://mlsanity.com/paper/2507.04377/) (2025)
- [Is neural semantic parsing good at ellipsis resolution, or isn't it?](https://mlsanity.com/paper/2508.00121/) (2025)
- [Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task](https://mlsanity.com/paper/2605.31480/) (2026)

## Abstract (arXiv)

The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing with a focus on semantic parsing and text generation. Currently, we witness an excellent performance of neural parsers and generators on the PMB. This might suggest that such semantic processing tasks have by and large been solved. We argue that this is not the case and that performance scores from the past on the PMB are inflated by non-optimal data splits and test sets that are too easy. In response, we introduce several changes. First, instead of the prior random split, we propose a more systematic splitting approach to improve the reliability of the standard test data. Second, except for the standard test set, we also propose two challenge sets: one with longer texts including discourse structure, and one that addresses compositional generalization. We evaluate five neural models for semantic parsing and meaning-to-text generation. Our results show that model performance declines (in some cases dramatically) on the challenge sets, revealing the limitations of neural models when confronting such challenges.