mlsanityData to 7 Oct 2026

Evaluating the Morphosyntactic Well-formedness of Generated Texts

arXiv 2103.16590 ↗First posted 30 Mar 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L'AMBRE -- a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 59 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

22 of 59 references.

References · 22

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  4. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 269
  5. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  6. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  7. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  8. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  9. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  10. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  11. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  12. 2018UniMorph 2.0: Universal Morphologycited by 40
10 more
  1. 2016How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairscited by 39
  2. 2019Pushing the Limits of Low-Resource Morphological Inflectioncited by 29
  3. 2019Grammatical Error Correction in Low-Resource Scenarioscited by 28
  4. 2020From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?cited by 26
  5. 2020Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluationcited by 22
  6. 2016There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correctioncited by 18
  7. 2018Marrying Universal Dependencies and Universal Morphologycited by 11
  8. 2020Automatic Extraction of Rules Governing Morphological Agreementcited by 9
  9. 2019A Latent Morphology Model for Open-Vocabulary Neural Machine Translationcited by 8
  10. 2017Referenceless Quality Estimation for Natural Language Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 1; 2024: 0; 2025: 1; 2026: 1.

8 papers.

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  3. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 23
  4. 2021Few-shot Controllable Style Transfer for Low-Resource Multilingual Settingscited by 20
  5. 2022CaMEL: Case Marker Extraction without Labelscited by 3
  6. 2022AUTOLEX: An Automatic Framework for Linguistic Explorationcited by 3
  7. 2021Minor changes make a difference: a case study on the consistency of UD-based dependency parserscited by 2
  8. 2026A Universal Vibe? Finding and Controlling Language-Agnostic Informal Register with SAEscited by 0