From Words to Worlds: Compositionality for Cognitive Architectures
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are very performant connectionist systems, but do they exhibit more compositionality? More importantly, is that part of why they perform so well? We present empirical analyses across four LLM families (12 models) and three task categories, including a novel task introduced below. Our findings reveal a nuanced relationship in learning of compositional strategies by LLMs -- while scaling enhances compositional abilities, instruction tuning often has a reverse effect. Such disparity brings forth some open issues regarding the development and improvement of large language models in alignment with human cognitive capacities.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 18 Jul 2024.
Builds on
34 of 58 references.
References · 34
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2023Mistral 7Bcited by 3,164
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
- 2023Instruction Tuning with GPT-4cited by 686
- 2023The Falcon Series of Open Language Modelscited by 535
22 more
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
- 2023Cognitive Architectures for Language Agentscited by 429
- 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
- 2020Calibration of Pre-trained Transformerscited by 315
- 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
- 2019Compositionality decomposed: how do neural networks generalise?cited by 265
- 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
- 2022Compositional Semantic Parsing with Large Language Modelscited by 95
- 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
- 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
- 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 66
- 2024Language models align with human judgments on key grammatical constructionscited by 49
- 2024Understanding and Patching Compositional Reasoning in LLMscited by 46
- 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
- 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
- 2023Instruction-tuning Aligns LLMs to the Human Braincited by 40
- 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
- 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
- 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
- 2023LLM Cognitive Judgements Differ From Humancited by 8
- 2023Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalizationcited by 7
- 2024SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Datacited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.