# Linguistic generalization and compositionality in modern artificial neural networks

arXiv: https://arxiv.org/abs/1904.00157 · HTML: https://mlsanity.com/paper/1904.00157/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2019.

## Built on by (most cited first, 30 of 58)

- [ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilities](https://mlsanity.com/paper/2407.01525/) (2024)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [Disentangling Syntax and Semantics in the Brain with Deep Networks](https://mlsanity.com/paper/2103.01620/) (2021)
- [Joint processing of linguistic properties in brains and language models](https://mlsanity.com/paper/2212.08094/) (2022)
- [Program Transfer for Answering Complex Questions over Knowledge Bases](https://mlsanity.com/paper/2110.05743/) (2021)
- [Emergent Communication: Generalization and Overfitting in Lewis Games](https://mlsanity.com/paper/2209.15342/) (2022)
- [Automated scholarly paper review: Concepts, technologies, and challenges](https://mlsanity.com/paper/2111.07533/) (2021)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models](https://mlsanity.com/paper/2310.15941/) (2023)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [Large language models can segment narrative events similarly to humans](https://mlsanity.com/paper/2301.10297/) (2023)
- [Geometric Signatures of Compositionality Across a Language Model's Lifetime](https://mlsanity.com/paper/2410.01444/) (2024)
- [Exploring Transitivity in Neural NLI Models through Veridicality](https://mlsanity.com/paper/2101.10713/) (2021)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [Improving Systematic Generalization Through Modularity and Augmentation](https://mlsanity.com/paper/2202.10745/) (2022)
- [Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task](https://mlsanity.com/paper/2204.06889/) (2022)
- [What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure](https://mlsanity.com/paper/2302.12239/) (2023)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off](https://mlsanity.com/paper/2301.13083/) (2023)
- [Grammar-Based Grounded Lexicon Learning](https://mlsanity.com/paper/2202.08806/) (2022)
- [Long-range and hierarchical language predictions in brains and algorithms](https://mlsanity.com/paper/2111.14232/) (2021)
- [A systematic framework for generating novel experimental hypotheses from language models](https://mlsanity.com/paper/2408.05086/) (2024)
- [Defending Compositionality in Emergent Languages](https://mlsanity.com/paper/2206.04751/) (2022)
- [SyGNS: A Systematic Generalization Testbed Based on Natural Language Semantics](https://mlsanity.com/paper/2106.01077/) (2021)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Schr"odinger's Tree -- On Syntax and Neural Language Models](https://mlsanity.com/paper/2110.08887/) (2021)
- [Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT](https://mlsanity.com/paper/2209.10538/) (2022)

## Abstract (arXiv)

In the last decade, deep artificial neural networks have achieved astounding performance in many natural language processing tasks. Given the high productivity of language, these models must possess effective generalization abilities. It is widely assumed that humans handle linguistic productivity by means of algebraic compositional rules: Are deep networks similarly compositional? After reviewing the main innovations characterizing current deep language processing networks, I discuss a set of studies suggesting that deep networks are capable of subtle grammar-dependent generalizations, but also that they do not rely on systematic compositional rules. I argue that the intriguing behaviour of these devices (still awaiting a full understanding) should be of interest to linguists and cognitive scientists, as it offers a new perspective on possible computational strategies to deal with linguistic productivity beyond rule-based compositionality, and it might lead to new insights into the less systematic generalization patterns that also appear in natural language.