# Holmes: A Benchmark to Assess the Linguistic Competence of Language Models

arXiv: https://arxiv.org/abs/2404.18923 · HTML: https://mlsanity.com/paper/2404.18923/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 146 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 30 of 79)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)

## Built on by (most cited first, 26 of 26)

- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench](https://mlsanity.com/paper/2407.13696/) (2024)
- [Linguini: A benchmark for language-agnostic linguistic reasoning](https://mlsanity.com/paper/2409.12126/) (2024)
- [Triangulating LLM Progress through Benchmarks, Games, and Cognitive Tests](https://mlsanity.com/paper/2502.14359/) (2025)
- [Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry](https://mlsanity.com/paper/2601.22588/) (2026)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence](https://mlsanity.com/paper/2411.07533/) (2024)
- [Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks](https://mlsanity.com/paper/2601.03448/) (2026)
- [FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation](https://mlsanity.com/paper/2504.17311/) (2025)
- [ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs](https://mlsanity.com/paper/2603.26516/) (2026)
- [Layered Insights: Generalizable Analysis of Authorial Style by Leveraging All Transformer Layers](https://mlsanity.com/paper/2503.00958/) (2025)
- [Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean](https://mlsanity.com/paper/2506.01237/) (2025)
- [Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not apply](https://mlsanity.com/paper/2509.01606/) (2025)
- [Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect](https://mlsanity.com/paper/2509.12065/) (2025)
- [Statistical Deficiency for Task Inclusion Estimation](https://mlsanity.com/paper/2503.05491/) (2025)
- [Instructions Shape Production of Language, not Processing](https://mlsanity.com/paper/2605.11206/) (2026)
- [A Pipeline to Assess Merging Methods via Behavior and Internals](https://mlsanity.com/paper/2509.19476/) (2025)
- [Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs](https://mlsanity.com/paper/2510.05278/) (2025)
- [Type and Complexity Signals in Multilingual Question Representations](https://mlsanity.com/paper/2510.06304/) (2025)
- [Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responses](https://mlsanity.com/paper/2511.23041/) (2025)
- [Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages](https://mlsanity.com/paper/2602.02182/) (2026)
- [Linguistic properties and model scale in brain encoding: from small to compressed language models](https://mlsanity.com/paper/2602.07547/) (2026)
- [Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models](https://mlsanity.com/paper/2602.20966/) (2026)
- [Controlled Paraphrase Geometry in Sentence Embedding Space: Local Manifold Modeling and Latent Probing](https://mlsanity.com/paper/2605.01073/) (2026)
- [AIMO Interpretability Challenge](https://mlsanity.com/paper/2607.13899/) (2026)
- [Bayesian Fine-tuning Yields Language Models that are as Bayesian as their Beliefs Allow](https://mlsanity.com/paper/2610.00679/) (2026)

## Abstract (arXiv)

We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence - their unconscious understanding of linguistic phenomena. Specifically, we use classifier-based probing to examine LMs' internal representations regarding distinct linguistic phenomena (e.g., part-of-speech tagging). As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities, such as following instructions in prompting-based evaluations. Composing Holmes, we review over 270 probing studies and include more than 200 datasets to assess syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing over 50 LMs reveals that, aligned with known trends, their linguistic competence correlates with model size. However, surprisingly, model architecture and instruction tuning also significantly influence performance, particularly in morphology and syntax. Finally, we propose FlashHolmes, a streamlined version that reduces the computation load while maintaining high-ranking precision.