# Dissociating language and thought in large language models

arXiv: https://arxiv.org/abs/2301.06627 · HTML: https://mlsanity.com/paper/2301.06627/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 127 of its 424 references that are in the map.
- First posted on arXiv 16 Jan 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 127)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)

## Built on by (most cited first, 29 of 29)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [A Vision Check-up for Language Models](https://mlsanity.com/paper/2401.01862/) (2024)
- [I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench](https://mlsanity.com/paper/2401.17882/) (2024)
- [Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference](https://mlsanity.com/paper/2404.14883/) (2024)
- [A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences](https://mlsanity.com/paper/2406.11341/) (2024)
- [Knowledge Engineering using Large Language Models](https://mlsanity.com/paper/2310.00637/) (2023)
- [Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding](https://mlsanity.com/paper/2310.18783/) (2023)
- [Learning a Structural Causal Model for Intuition Reasoning in Conversation](https://mlsanity.com/paper/2305.17727/) (2023)
- [Sub-goal Distillation: A Method to Improve Small Language Agents](https://mlsanity.com/paper/2405.02749/) (2024)
- [Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning](https://mlsanity.com/paper/2312.05571/) (2023)
- [Stars, Stripes, and Silicon: Unravelling the ChatGPT's All-American, Monochrome, Cis-centric Bias](https://mlsanity.com/paper/2410.13868/) (2024)
- [CLEVR-POC: Reasoning-Intensive Visual Question Answering in Partially Observable Environments](https://mlsanity.com/paper/2403.03203/) (2024)
- [Can formal argumentative reasoning enhance LLMs performances?](https://mlsanity.com/paper/2405.13036/) (2024)
- [CRAFT: Grounded Multi-Agent Coordination Under Partial Information](https://mlsanity.com/paper/2603.25268/) (2026)
- [Sparse Autoencoders Map Brain-LLM Alignment onto Cortical Semantic Topography](https://mlsanity.com/paper/2605.23035/) (2026)
- [Attention, not scale, drives human-AI alignment in multimodal language prediction](https://mlsanity.com/paper/2308.06035/) (2023)
- [Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMs](https://mlsanity.com/paper/2605.23039/) (2026)
- [Model Collapse as Cultural Evolution](https://mlsanity.com/paper/2605.23054/) (2026)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks](https://mlsanity.com/paper/2609.34187/) (2026)
- [The Presupposition Problem in Representation Genesis](https://mlsanity.com/paper/2603.21745/) (2026)
- [Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?](https://mlsanity.com/paper/2604.17930/) (2026)
- [A Lightweight Multi-Agent Framework for Automated Concrete Barrier Design](https://mlsanity.com/paper/2606.12040/) (2026)
- [Thinking effort aligns between humans and reasoning models in abductive reasoning](https://mlsanity.com/paper/2609.01867/) (2026)
- [Cognitive Expert Language Models Better Align with the Corresponding Brain Systems](https://mlsanity.com/paper/2609.36239/) (2026)
- [Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction](https://mlsanity.com/paper/2610.00840/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction between formal linguistic competence -- knowledge of linguistic rules and patterns -- and functional linguistic competence -- understanding and using language in the world. We ground this distinction in human neuroscience, which has shown that formal and functional competence rely on different neural mechanisms. Although LLMs are surprisingly good at formal competence, their performance on functional competence tasks remains spotty and often requires specialized fine-tuning and/or coupling with external modules. We posit that models that use language in human-like ways would need to master both of these competence types, which, in turn, could require the emergence of mechanisms specialized for formal linguistic competence, distinct from functional competence.