# Language models align with human judgments on key grammatical constructions

arXiv: https://arxiv.org/abs/2402.01676 · HTML: https://mlsanity.com/paper/2402.01676/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 11 references that are in the map.
- First posted on arXiv 19 Jan 2024.

## Builds on (most cited first, 4 of 4)

- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Prompting is not a substitute for probability measurements in large language models](https://mlsanity.com/paper/2305.13264/) (2023)

## Built on by (most cited first, 30 of 49)

- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Auxiliary task demands mask the capabilities of smaller language models](https://mlsanity.com/paper/2404.02418/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://mlsanity.com/paper/2402.12560/) (2024)
- [Language Models Fail to Introspect About Their Knowledge of Language](https://mlsanity.com/paper/2503.07513/) (2025)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models](https://mlsanity.com/paper/2403.14859/) (2024)
- [What Can String Probability Tell Us About Grammaticality?](https://mlsanity.com/paper/2510.16227/) (2025)
- [Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference](https://mlsanity.com/paper/2404.14883/) (2024)
- [Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs](https://mlsanity.com/paper/2502.13195/) (2025)
- [TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2506.13487/) (2025)
- [How Syntax Specialization Emerges in Language Models](https://mlsanity.com/paper/2505.19548/) (2025)
- [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025)
- [Large Language Model probabilities cannot distinguish between possible and impossible language](https://mlsanity.com/paper/2509.15114/) (2025)
- [Post-training makes large language models less human-like](https://mlsanity.com/paper/2605.07632/) (2026)
- [Anthropocentric bias in language model evaluation](https://mlsanity.com/paper/2407.03859/) (2024)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibility](https://mlsanity.com/paper/2503.17579/) (2025)
- [How important is language for human-like intelligence?](https://mlsanity.com/paper/2509.15560/) (2025)
- [Comparing Human and Language Models Sentence Processing Difficulties on Complex Structures](https://mlsanity.com/paper/2510.07141/) (2025)
- [Using Shapley interactions to understand how models use structure](https://mlsanity.com/paper/2403.13106/) (2024)
- [From Words to Worlds: Compositionality for Cognitive Architectures](https://mlsanity.com/paper/2407.13419/) (2024)
- [For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependencies](https://mlsanity.com/paper/2505.09005/) (2025)
- [Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs](https://mlsanity.com/paper/2507.22286/) (2025)
- [Representational Similarity Alignment of Figurative Language in Humans and Large Language Models](https://mlsanity.com/paper/2601.09041/) (2026)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Knowledge Model Prompting Increases LLM Performance on Planning Tasks](https://mlsanity.com/paper/2602.03900/) (2026)
- [Predicting Sentence Acceptability Judgments in Multimodal Contexts](https://mlsanity.com/paper/2602.20918/) (2026)

## Abstract (arXiv)

Do large language models (LLMs) make human-like linguistic generalizations? Dentella et al. (2023) ("DGL") prompt several LLMs ("Is the following sentence grammatically correct in English?") to elicit grammaticality judgments of 80 English sentences, concluding that LLMs demonstrate a "yes-response bias" and a "failure to distinguish grammatical from ungrammatical sentences". We re-evaluate LLM performance using well-established practices and find that DGL's data in fact provide evidence for just how well LLMs capture human behaviors. Models not only achieve high accuracy overall, but also capture fine-grained variation in human linguistic judgments.