mlsanityData to 7 Oct 2026

Language models align with human judgments on key grammatical constructions

arXiv 2402.01676 ↗First posted 19 Jan 2024cs.CL cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Do large language models (LLMs) make human-like linguistic generalizations? Dentella et al. (2023) ("DGL") prompt several LLMs ("Is the following sentence grammatically correct in English?") to elicit grammaticality judgments of 80 English sentences, concluding that LLMs demonstrate a "yes-response bias" and a "failure to distinguish grammatical from ungrammatical sentences". We re-evaluate LLM performance using well-established practices and find that DGL's data in fact provide evidence for just how well LLMs capture human behaviors. Models not only achieve high accuracy overall, but also capture fine-grained variation in human linguistic judgments.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 4 of its 11 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

4 of 11 references.

References · 4

  1. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  2. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  3. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  4. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 25; 2026: 16.

49 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  2. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  3. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  4. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  5. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  6. 2025Large Language Models Are Human-Like Internallycited by 28
  7. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  8. 2024Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Modelscited by 24
  9. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  10. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  11. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  12. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
37 more
  1. 2025How Syntax Specialization Emerges in Language Modelscited by 7
  2. 2025Large Language Model probabilities cannot distinguish between possible and impossible languagecited by 5
  3. 2024Anthropocentric bias in language model evaluationcited by 4
  4. 2025Probing Syntax in Large Language Models: Successes and Remaining Challengescited by 4
  5. 2026Post-training makes large language models less human-likecited by 4
  6. 2025Trick or Neat: Adversarial Ambiguity and Language Model Evaluationcited by 3
  7. 2025The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theoriescited by 2
  8. 2025Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibilitycited by 2
  9. 2025How important is language for human-like intelligence?cited by 2
  10. 2025Comparing Human and Language Models Sentence Processing Difficulties on Complex Structurescited by 2
  11. 2024Using Shapley interactions to understand how models use structurecited by 1
  12. 2024From Words to Worlds: Compositionality for Cognitive Architecturescited by 1
  13. 2025For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependenciescited by 1
  14. 2025Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMscited by 1
  15. 2026Can LLMs interpret figurative language as humans do?: surface-level vs representational similaritycited by 1
  16. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  17. 2026Knowledge Model Prompting Increases LLM Performance on Planning Taskscited by 1
  18. 2026Predicting Sentence Acceptability Judgments in Multimodal Contextscited by 1
  19. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  20. 2026Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMscited by 1
  21. 2025Do Large Language Models know who did what to whom?cited by 0
  22. 2025BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Annotations and Rationale Indicatorscited by 0
  23. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  24. 2025Quantifier Scope Interpretation in Language Learners and LLMscited by 0
  25. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  26. 2025Large language models have learned to use languagecited by 0
  27. 2025Large language models are not about natural languagecited by 0
  28. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  29. 2026Are Language Models Models?cited by 0
  30. 2026Extending Minimal Pairs with Ordinal Surprisal Curves and Entropy Across Applied Domainscited by 0
  31. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  32. 2026You Can't Fight in Here! This is BBS!cited by 0
  33. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  34. 2026Linear representations of grammaticality in neural language modelscited by 0
  35. 2026On the use of foundation models in cognitive sciencecited by 0
  36. 2026LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoningcited by 0
  37. 2026Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  3. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  4. 2024Dissecting Human and LLM Preferences
  5. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  6. 2026Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
  7. 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings
  8. 2026LLMs Can Better Capture Human Judgments--With the Right Prompts