mlsanityData to 14 Oct 2026

MELA: Multilingual Evaluation of Linguistic Acceptability

arXiv 2311.09033 ↗First posted 15 Nov 2023cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability -- MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language -- Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. Our data is available at https://github.com/sjtu-compling/MELA.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 62 of its 98 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

62 of 98 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  14. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  15. 2018Neural Network Acceptability Judgmentscited by 958
  16. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  18. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  19. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  20. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  21. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  22. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  23. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  24. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  25. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  26. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  27. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  28. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  29. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  30. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  31. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  32. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  33. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  34. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  35. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  36. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  37. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  38. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  3. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
  4. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  5. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  6. 2025SynthDetoxM: Modern LLMs are Few-Shot Parallel Detoxification Data Annotatorscited by 4
  7. 2025DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errorscited by 3
  8. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
  9. 2026M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Modelscited by 1
  10. 2025COLA-GEC: A Bidirectional Framework for Enhancing Grammatical Acceptability and Error Correctioncited by 0
  11. 2025From Black Box to Transparency: Enhancing Automated Interpreting Assessment with Explainable AI in College Classroomscited by 0
  12. 2025MTQ-Eval: Multilingual Text Quality Evaluation for Language Modelscited by 0
2 more
  1. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  2. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0