Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages
Not on the map. Why?
Abstract (arXiv)
Despite the fact that Transformers perform well in NLP tasks, recent studies suggest that self-attention is theoretically limited in learning even some regular and context-free languages. These findings motivated us to think about their implications in modeling natural language, which is hypothesized to be mildly context-sensitive. We test the Transformer's ability to learn mildly context-sensitive languages of varying complexities, and find that they generalize well to unseen in-distribution data, but their ability to extrapolate to longer strings is worse than that of LSTMs. Our analyses show that the learned self-attention patterns and representations modeled dependency relations and demonstrated counting behavior, which may have helped the models solve the languages.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 98 references that are in the map. First posted on arXiv 2 Sep 2023.
Builds on
36 of 98 references.
References · 36
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Layer Normalizationcited by 5,669
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2018Self-Attention with Relative Position Representationscited by 1,120
- 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2019Are Sixteen Heads Really Better than One?cited by 816
24 more
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2021A Survey of Transformerscited by 401
- 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
- 2022Neural Networks and the Chomsky Hierarchycited by 217
- 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
- 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
- 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
- 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
- 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
- 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
- 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
- 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
- 2019Language Modeling with Deep Transformerscited by 96
- 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
- 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
- 2022A Logic for Expressing Log-Precision Transformerscited by 67
- 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
- 2020RNNs can generate bounded hierarchical languages with optimal memorycited by 59
- 2019Sequential Neural Networks as Automatacited by 54
- 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
- 2020The EOS Decision and Length Extrapolationcited by 46
- 2018On Evaluating the Generalization of LSTM Models in Formal Languagescited by 17
- 2021Formal Language Theory Meets Modern NLPcited by 9
- 2022Discontinuous Constituency and BERT: A Case Study of Dutchcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.