# Evaluating the Morphosyntactic Well-formedness of Generated Texts

arXiv: https://arxiv.org/abs/2103.16590 · HTML: https://mlsanity.com/paper/2103.16590/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbations](https://mlsanity.com/paper/2005.04364/) (2020)
- [Cross-Linguistic Syntactic Evaluation of Word Prediction Models](https://mlsanity.com/paper/2005.00187/) (2020)
- [UniMorph 2.0: Universal Morphology](https://mlsanity.com/paper/1810.11101/) (2018)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://mlsanity.com/paper/1612.04629/) (2016)
- [Pushing the Limits of Low-Resource Morphological Inflection](https://mlsanity.com/paper/1908.05838/) (2019)
- [Grammatical Error Correction in Low-Resource Scenarios](https://mlsanity.com/paper/1910.00353/) (2019)
- [From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?](https://mlsanity.com/paper/2005.01330/) (2020)
- [Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation](https://mlsanity.com/paper/2005.10716/) (2020)
- [There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction](https://mlsanity.com/paper/1610.02124/) (2016)
- [Marrying Universal Dependencies and Universal Morphology](https://mlsanity.com/paper/1810.06743/) (2018)
- [Automatic Extraction of Rules Governing Morphological Agreement](https://mlsanity.com/paper/2010.01160/) (2020)
- [A Latent Morphology Model for Open-Vocabulary Neural Machine Translation](https://mlsanity.com/paper/1910.13890/) (2019)
- [Referenceless Quality Estimation for Natural Language Generation](https://mlsanity.com/paper/1708.01759/) (2017)

## Built on by (most cited first, 8 of 8)

- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks](https://mlsanity.com/paper/2310.19740/) (2023)
- [Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings](https://mlsanity.com/paper/2110.07385/) (2021)
- [CaMEL: Case Marker Extraction without Labels](https://mlsanity.com/paper/2203.10010/) (2022)
- [AUTOLEX: An Automatic Framework for Linguistic Exploration](https://mlsanity.com/paper/2203.13901/) (2022)
- [Minor changes make a difference: a case study on the consistency of UD-based dependency parsers](https://mlsanity.com/paper/2111.15413/) (2021)
- [A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEs](https://mlsanity.com/paper/2603.26236/) (2026)

## Abstract (arXiv)

Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L'AMBRE -- a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.