# Do large language models resemble humans in language use?

arXiv: https://arxiv.org/abs/2303.08014 · HTML: https://mlsanity.com/paper/2303.08014/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 81 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models](https://mlsanity.com/paper/1909.10579/) (2019)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [What's in an accent? The impact of accented synthetic speech on lexical choice in human-machine dialogue](https://mlsanity.com/paper/1907.11146/) (2019)

## Built on by (most cited first, 30 of 34)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters](https://mlsanity.com/paper/2312.17476/) (2023)
- [CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation](https://mlsanity.com/paper/2404.08806/) (2024)
- [LLMs syntactically adapt their language use to their conversational partner](https://mlsanity.com/paper/2503.07457/) (2025)
- [AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers](https://mlsanity.com/paper/2410.01824/) (2024)
- [AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays](https://mlsanity.com/paper/2304.14276/) (2023)
- [LLM Cognitive Judgements Differ From Human](https://mlsanity.com/paper/2307.11787/) (2023)
- [Unveiling Language Competence Neurons: A Psycholinguistic Approach to Model Interpretability](https://mlsanity.com/paper/2409.15827/) (2024)
- [HLB: Benchmarking LLMs' Humanlikeness in Language Use](https://mlsanity.com/paper/2409.15890/) (2024)
- [How many words does ChatGPT know? The answer is ChatWords](https://mlsanity.com/paper/2309.16777/) (2023)
- [Does Conceptual Representation Require Embodiment? Insights From Large Language Models](https://mlsanity.com/paper/2305.19103/) (2023)
- [Grammaticality Representation in ChatGPT as Compared to Linguists and Laypeople](https://mlsanity.com/paper/2406.11116/) (2024)
- [Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Features](https://mlsanity.com/paper/2407.03646/) (2024)
- [Topic-Conversation Relevance (TCR) Dataset and Benchmarks](https://mlsanity.com/paper/2411.00038/) (2024)
- [Speaker effects in language comprehension: An integrative model of language and speaker processing](https://mlsanity.com/paper/2412.07238/) (2024)
- [Linguistics and Human Brain: A Perspective of Computational Neuroscience](https://mlsanity.com/paper/2602.08275/) (2026)
- [Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing](https://mlsanity.com/paper/2501.09127/) (2025)
- [Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?](https://mlsanity.com/paper/2503.10838/) (2025)
- [Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism](https://mlsanity.com/paper/2511.10045/) (2025)
- [AI generates well-liked but templatic empathic responses](https://mlsanity.com/paper/2604.08479/) (2026)
- [The Tyranny of Possibilities in the Design of Task-Oriented LLM Systems: A Scoping Survey](https://mlsanity.com/paper/2312.17601/) (2023)
- [Can AI mimic the human ability to define neologisms?](https://mlsanity.com/paper/2502.14900/) (2025)
- [Understanding the Impact of Artificial Intelligence in Academic Writing: Metadata to the Rescue](https://mlsanity.com/paper/2502.16713/) (2025)
- [BAMBI: Developing Baby Language Models for Italian](https://mlsanity.com/paper/2503.09481/) (2025)
- [Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMs](https://mlsanity.com/paper/2509.14456/) (2025)
- [Representational Similarity Alignment of Figurative Language in Humans and Large Language Models](https://mlsanity.com/paper/2601.09041/) (2026)
- [MacBehaviour: An R package for behavioural experimentation on large language models](https://mlsanity.com/paper/2405.07495/) (2024)
- [The Career Interests of Large Language Models](https://mlsanity.com/paper/2407.08564/) (2024)
- [A funny companion: Distinct neural responses to AI- versus human-attributed humor](https://mlsanity.com/paper/2509.10847/) (2025)

## Abstract (arXiv)

Large language models (LLMs) such as ChatGPT and Vicuna have shown remarkable capacities in comprehending and producing language. However, their internal workings remain a black box, and it is unclear whether LLMs and chatbots can develop humanlike characteristics in language use. Cognitive scientists have devised many experiments that probe, and have made great progress in explaining, how people comprehend and produce language. We subjected ChatGPT and Vicuna to 12 of these experiments ranging from sounds to dialogue, preregistered and with 1000 runs (i.e., iterations) per experiment. ChatGPT and Vicuna replicated the human pattern of language use in 10 and 7 out of the 12 experiments, respectively. The models associated unfamiliar words with different meanings depending on their forms, continued to access recently encountered meanings of ambiguous words, reused recent sentence structures, attributed causality as a function of verb semantics, and accessed different meanings and retrieved different words depending on an interlocutor's identity. In addition, ChatGPT, but not Vicuna, nonliterally interpreted implausible sentences that were likely to have been corrupted by noise, drew reasonable inferences, and overlooked semantic fallacies in a sentence. Finally, unlike humans, neither model preferred using shorter words to convey less informative content, nor did they use context to resolve syntactic ambiguities. We discuss how these convergences and divergences may result from the transformer architecture. Overall, these experiments demonstrate that LLMs such as ChatGPT (and Vicuna to a lesser extent) are humanlike in many aspects of human language processing.