mlsanityData to 14 Oct 2026

A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese

arXiv 2411.06096 ↗First posted 9 Nov 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the textitBa construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that textscAnaphor, textscQuantifiers, and textscEllipsis in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 26 of its 68 references that are in the map. First posted on arXiv 9 Nov 2024.

Builds on

26 of 68 references.

References · 26

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  8. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  10. 2018A Simple Method for Commonsense Reasoningcited by 223
  11. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  12. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
14 more
  1. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  2. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  3. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  4. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  5. 2023Prompting is not a substitute for probability measurements in large language modelscited by 44
  6. 2023Language acquisition: do children and language models follow similar learning stages?cited by 41
  7. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  8. 2020How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Contextcited by 26
  9. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  10. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  11. 2024Release of Pre-Trained Models for the Japanese Languagecited by 22
  12. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  13. 2022SLING: Sino Linguistic Evaluation of Large Language Modelscited by 15
  14. 2024What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Lengthcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 6.

13 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  3. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 8
  4. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  5. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  6. 2025HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuningcited by 1
  7. 2026Linear representations of grammaticality in neural language modelscited by 1
  8. 2025Understanding Syntactic Generalization in Structure-inducing Language Modelscited by 0
  9. 2026Left-right asymmetry in predicting brain activity from LLMs' representations emerges with their formal linguistic competencecited by 0
  10. 2026The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinesecited by 0
  11. 2026Objective vs. Search: Decomposing What Makes a Good Tokenisercited by 0
  12. 2026Beetle: A Bilingual Model Suite for Modelling Second-Language Processingcited by 0
1 more
  1. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0