mlsanityData to 14 Oct 2026

Holmes: A Benchmark to Assess the Linguistic Competence of Language Models

arXiv 2404.18923 ↗First posted 29 Apr 2024cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence - their unconscious understanding of linguistic phenomena. Specifically, we use classifier-based probing to examine LMs' internal representations regarding distinct linguistic phenomena (e.g., part-of-speech tagging). As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities, such as following instructions in prompting-based evaluations. Composing Holmes, we review over 270 probing studies and include more than 200 datasets to assess syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing over 50 LMs reveals that, aligned with known trends, their linguistic competence correlates with model size. However, surprisingly, model architecture and instruction tuning also significantly influence performance, particularly in morphology and syntax. Finally, we propose FlashHolmes, a streamlined version that reduces the computation load while maintaining high-ranking precision.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 79 of its 146 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

79 of 146 references.

References · 79

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  10. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  13. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  14. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  15. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  16. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  17. 2019Designing and Interpreting Probes with Control Taskscited by 640
  18. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  19. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  20. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  21. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  22. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  23. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  24. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  25. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  26. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 232
  27. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  28. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  29. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  30. 2020How Context Affects Language Models' Factual Predictionscited by 181
  31. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  32. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  33. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  34. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  35. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  36. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  37. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  38. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025Large Language Models Are Human-Like Internallycited by 29
  2. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
  3. 2024Linguini: A benchmark for language-agnostic linguistic reasoningcited by 17
  4. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 12
  5. 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
  6. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  7. 2024Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competencecited by 4
  8. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  9. 2025FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluationcited by 3
  10. 2026ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMscited by 3
  11. 2025Layered Insights: Generalizable Analysis of Authorial Style by Leveraging All Transformer Layerscited by 2
  12. 2025Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Koreancited by 2
14 more
  1. 2025Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not applycited by 2
  2. 2025Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspectcited by 2
  3. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1
  4. 2026Instructions Shape Production of Language, not Processingcited by 1
  5. 2025A Pipeline to Assess Merging Methods via Behavior and Internalscited by 0
  6. 2025Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEscited by 0
  7. 2025Type and Complexity Signals in Multilingual Question Representationscited by 0
  8. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  9. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  10. 2026Linguistic properties and model scale in brain encoding: from small to compressed language modelscited by 0
  11. 2026Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Modelscited by 0
  12. 2026Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probingcited by 0
  13. 2026AIMO Interpretability Challengecited by 0
  14. 2026Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allowcited by 0