mlsanityData to 14 Oct 2026

CLiMP: A Benchmark for Chinese Language Model Evaluation

arXiv 2101.11131 ↗First posted 26 Jan 2021cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Linguistically informed analyses of language models (LMs) contribute to the understanding and improvement of these models. Here, we introduce the corpus of Chinese linguistic minimal pairs (CLiMP), which can be used to investigate what knowledge Chinese LMs acquire. CLiMP consists of sets of 1,000 minimal pairs (MPs) for 16 syntactic contrasts in Mandarin, covering 9 major Mandarin linguistic phenomena. The MPs are semi-automatically generated, and human agreement with the labels in CLiMP is 95.8%. We evaluated 11 different LMs on CLiMP, covering n-grams, LSTMs, and Chinese BERT. We find that classifier-noun agreement and verb complement selection are the phenomena that models generally perform best at. However, models struggle the most with the ba construction, binding, and filler-gap dependencies. Overall, Chinese BERT achieves an 81.8% average accuracy, while the performances of LSTMs and 5-grams are only moderately above chance level.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 26 Jan 2021.

Builds on

19 of 28 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2018Neural Network Acceptability Judgmentscited by 958
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  11. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  12. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
7 more
  1. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  2. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  3. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  4. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  5. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  6. 2018Assessing Composition in Sentence Vector Representationscited by 26
  7. 2018Can LSTM Learn to Capture Agreement? The Case of Basquecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 3; 2023: 7; 2024: 8; 2025: 19; 2026: 6.

46 papers.

Sort:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  3. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  4. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 19
  5. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  6. 2023ScandEval: A Benchmark for Scandinavian Natural Language Processingcited by 16
  7. 2021On the Limits of Minimal Pairs in Contrastive Evaluationcited by 15
  8. 2022SLING: Sino Linguistic Evaluation of Large Language Modelscited by 15
  9. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  10. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
  11. 2025LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encodercited by 14
  12. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
34 more
  1. 2023WYWEB: A NLP Evaluation Benchmark For Classical Chinesecited by 12
  2. 2025XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairscited by 12
  3. 2023JCoLA: Japanese Corpus of Linguistic Acceptabilitycited by 11
  4. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  5. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  6. 2025Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisitioncited by 10
  7. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  8. 2024From Human Judgements to Predictive Models: Unravelling Acceptability in Code-Mixed Sentencescited by 8
  9. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 8
  10. 2025Chengyu-Bench: Benchmarking Large Language Models for Chinese Idiom Understanding and Usecited by 8
  11. 2023Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Featurescited by 6
  12. 2021Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Modelscited by 5
  13. 2025Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representationscited by 5
  14. 2024Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competencecited by 4
  15. 2025Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?cited by 3
  16. 2022Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentationcited by 2
  17. 2024Quantum Transfer Learning for Acceptability Judgementscited by 2
  18. 2025Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive zijicited by 2
  19. 2025UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urducited by 2
  20. 2025QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgmentscited by 2
  21. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  22. 2025Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Settingcited by 2
  23. 2024Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Modelscited by 1
  24. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  25. 2026Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Modelscited by 1
  26. 2026M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Modelscited by 1
  27. 2023Revisiting Acceptability Judgementscited by 0
  28. 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
  29. 2025COLA-GEC: A Bidirectional Framework for Enhancing Grammatical Acceptability and Error Correctioncited by 0
  30. 2025textscCantoNLU: A benchmark for Cantonese natural language understandingcited by 0
  31. 2025Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgishcited by 0
  32. 2026Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italiancited by 0
  33. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  34. 2026BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Itemscited by 0