# Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation

arXiv: https://arxiv.org/abs/2306.00124 · HTML: https://mlsanity.com/paper/2306.00124/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data](https://mlsanity.com/paper/2203.08773/) (2022)
- [Graph Pre-training for AMR Parsing and Generation](https://mlsanity.com/paper/2203.07836/) (2022)
- [Rethinking Supervised Pre-training for Better Downstream Transferring](https://mlsanity.com/paper/2110.06014/) (2021)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Exploring Neural Methods for Parsing Discourse Representation Structures](https://mlsanity.com/paper/1810.12579/) (2018)
- [Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer](https://mlsanity.com/paper/2204.07549/) (2022)
- [Semantic Graph Parsing with Recurrent Neural Network DAG Grammars](https://mlsanity.com/paper/1910.00051/) (2019)
- [Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer](https://mlsanity.com/paper/2203.08552/) (2022)
- [DRTS Parsing with Structure-Aware Encoding and Decoding](https://mlsanity.com/paper/2005.06901/) (2020)
- [The Parallel Meaning Bank: A Framework for Semantically Annotating Multiple Languages](https://mlsanity.com/paper/2012.14854/) (2020)

## Built on by (most cited first, 10 of 10)

- [Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations](https://mlsanity.com/paper/2404.12698/) (2024)
- [PMB5: Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks](https://mlsanity.com/paper/2404.08354/) (2024)
- [KUNPENG: An Embodied Large Model for Intelligent Maritime](https://mlsanity.com/paper/2407.09048/) (2024)
- [MACT: Model-Agnostic Cross-Lingual Training for Discourse Representation Structure Parsing](https://mlsanity.com/paper/2406.01052/) (2024)
- [Scope-enhanced Compositional Semantic Parsing for DRT](https://mlsanity.com/paper/2407.01899/) (2024)
- [Retrieval-Augmented Semantic Parsing: Improving Generalization with Lexical Knowledge](https://mlsanity.com/paper/2412.10207/) (2024)
- [CogNarr Ecosystem: Facilitating Group Cognition at Scale](https://mlsanity.com/paper/2407.18945/) (2024)
- [Is neural semantic parsing good at ellipsis resolution, or isn't it?](https://mlsanity.com/paper/2508.00121/) (2025)
- [Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya](https://mlsanity.com/paper/2607.15209/) (2026)
- [Structural Automata for Neural Semantic Parsing](https://mlsanity.com/paper/2610.13896/) (2026)

## Abstract (arXiv)

Pre-trained language models (PLMs) have achieved great success in NLP and have recently been used for tasks in computational semantics. However, these tasks do not fully benefit from PLMs since meaning representations are not explicitly included in the pre-training stage. We introduce multilingual pre-trained language-meaning models based on Discourse Representation Structures (DRSs), including meaning representations besides natural language texts in the same model, and design a new strategy to reduce the gap between the pre-training and fine-tuning objectives. Since DRSs are language neutral, cross-lingual transfer learning is adopted to further improve the performance of non-English tasks. Automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks. Correlation analysis between automatic metrics and human judgements on the generation task further validates the effectiveness of our model. Human inspection reveals that out-of-vocabulary tokens are the main cause of erroneous results.