# The Unstoppable Rise of Computational Linguistics in Deep Learning

arXiv: https://arxiv.org/abs/2005.06420 · HTML: https://mlsanity.com/paper/2005.06420/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2020.

## Built on by (most cited first, 12 of 12)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systems](https://mlsanity.com/paper/2205.01128/) (2022)
- [HyperMixer: An MLP-based Low Cost Alternative to Transformers](https://mlsanity.com/paper/2203.03691/) (2022)
- [What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris](https://mlsanity.com/paper/2205.07750/) (2022)
- [On the Evolution of Syntactic Information Encoded by BERT's Contextualized Representations](https://mlsanity.com/paper/2101.11492/) (2021)
- [Transformers as Graph-to-Graph Models](https://mlsanity.com/paper/2310.17936/) (2023)
- [A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training time](https://mlsanity.com/paper/2206.02440/) (2022)
- [A Variational AutoEncoder for Transformers with Nonparametric Variational Information Bottleneck](https://mlsanity.com/paper/2207.13529/) (2022)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation](https://mlsanity.com/paper/2110.07002/) (2021)
- [GADePo: Graph-Assisted Declarative Pooling Transformers for Document-Level Relation Extraction](https://mlsanity.com/paper/2308.14423/) (2023)
- [Investigating Idiomaticity in Word Representations](https://mlsanity.com/paper/2411.02610/) (2024)

## Abstract (arXiv)

In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the challenges facing research in deep learning architectures for natural language understanding.