# Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models

arXiv: https://arxiv.org/abs/2510.04764 · HTML: https://mlsanity.com/paper/2510.04764/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty](https://mlsanity.com/paper/2308.02019/) (2023)
- [Trained on 100 million words and still in shape: BERT meets British National Corpus](https://mlsanity.com/paper/2303.09859/) (2023)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2404.06214/) (2024)
- [Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom](https://mlsanity.com/paper/2404.19509/) (2024)
- [LLMs Struggle to Reject False Presuppositions when Misinformation Stakes are High](https://mlsanity.com/paper/2505.22354/) (2025)
- [Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations](https://mlsanity.com/paper/2305.14618/) (2023)
- [Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2411.07474/) (2024)
- [Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions](https://mlsanity.com/paper/2506.08952/) (2025)
- [Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://mlsanity.com/paper/2403.12675/) (2024)
- [Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation](https://mlsanity.com/paper/2504.16060/) (2025)
- [Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding](https://mlsanity.com/paper/2507.15357/) (2025)
- [Experimental Pragmatics with Machines: Testing LLM Predictions for the Inferences of Plain and Embedded Disjunctions](https://mlsanity.com/paper/2405.05776/) (2024)
- [Pragmatic inference of scalar implicature by LLMs](https://mlsanity.com/paper/2408.06673/) (2024)
- [Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks?](https://mlsanity.com/paper/2506.11807/) (2025)

## Built on by (most cited first, 11 of 11)

- [Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models](https://mlsanity.com/paper/2604.07749/) (2026)
- [How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models](https://mlsanity.com/paper/2604.15873/) (2026)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [A retrospective analysis on the use of LLMs to study infant syntax learning](https://mlsanity.com/paper/2609.26539/) (2026)

## Abstract (arXiv)

Implicit meanings are integral to human communication, making it essential for language models to be capable of identifying and interpreting them. Grice (1975) proposed a set of conversational maxims that guide cooperative dialogue, noting that speakers may deliberately violate these principles to express meanings beyond literal words, and that listeners, in turn, recognize such violations to draw pragmatic inferences. Building on Surian et al. (1996)'s study of children's sensitivity to violations of Gricean maxims, we introduce a novel benchmark to test whether language models pretrained on less than 10M and less than 100M tokens can distinguish maxim-adhering from maxim-violating utterances. We compare these BabyLMs across five maxims and situate their performance relative to children and a Large Language Model (LLM) pretrained on 3T tokens. We find that overall, models trained on less than 100M tokens outperform those trained on less than 10M, yet fall short of child-level and LLM competence. Our results suggest that modest data increases improve some aspects of pragmatic behavior, leading to finer-grained differentiation between pragmatic dimensions.