What Makes Language Models Good-enough?
Not on the map. Why?
Abstract (arXiv)
Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing. We focus on the number of layers and self-attention heads in Transformers. We create a good-enough language processing (GELP) evaluation dataset (7,680 examples), which is designed to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. To annotate GELP, we first conduct a crowdsourcing experiment whose design follows prior psycholinguistic studies. Our model evaluation against the annotated GELP then reveals that the full model as well as models with fewer layers and/or self-attention heads exhibit a good-enough performance. This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 6 Jun 2024.
Builds on
25 of 48 references.
References · 25
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2015A large annotated corpus for learning natural language inferencecited by 1,837
- 2018Neural Network Acceptability Judgmentscited by 952
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
- 2018Annotation Artifacts in Natural Language Inference Datacited by 656
- 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
13 more
- 2018Targeted Syntactic Evaluation of Language Modelscited by 239
- 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
- 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
- 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
- 2018What Makes Reading Comprehension Questions Easier?cited by 59
- 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
- 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
- 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
- 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
- 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
- 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
- 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
- 2022Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systemscited by 7
Built on by
Citing papers per year: 2026: 2.
2 papers.