Construction Grammar and Language Models
Not on the map. Why?
Abstract (arXiv)
Recent progress in deep learning and natural language processing has given rise to powerful models that are primarily trained on a cloze-like task and show some evidence of having access to substantial linguistic information, including some constructional knowledge. This groundbreaking discovery presents an exciting opportunity for a synergistic relationship between computational methods and Construction Grammar research. In this chapter, we explore three distinct approaches to the interplay between computational methods and Construction Grammar: (i) computational methods for text analysis, (ii) computational Construction Grammar, and (iii) deep learning models, with a particular focus on language models. We touch upon the first two approaches as a contextual foundation for the use of computational methods before providing an accessible, yet comprehensive overview of deep learning models, which also addresses reservations construction grammarians may have. Additionally, we delve into experiments that explore the emergence of constructionally relevant information within these models while also examining the aspects of Construction Grammar that may pose challenges for these models. This chapter aims to foster collaboration between researchers in the fields of natural language processing and Construction Grammar. By doing so, we hope to pave the way for new insights and advancements in both these fields.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 90 references that are in the map. First posted on arXiv 25 Aug 2023.
Builds on
40 of 90 references.
References · 40
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
28 more
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
- 2018Colorless green recurrent networks dream hierarchicallycited by 213
- 2015Semi-supervised Sequence Learningcited by 198
- 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
- 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
- 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
- 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
- 2020Syntactic Structure from Deep Learningcited by 99
- 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
- 2019Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguationcited by 40
- 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
- 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
- 2020CxGBERT: BERT meets Construction Grammarcited by 32
- 2022Neural reality of argument structure constructionscited by 25
- 2020Parsing as Pretrainingcited by 19
- 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
- 2020Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledgecited by 18
- 2020What's in a Name? Are BERT Named Entity Representations just as Good for any other Name?cited by 17
- 2021LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyondcited by 17
- 2020The Unstoppable Rise of Computational Linguistics in Deep Learningcited by 12
- 2019Inducing Syntactic Trees from BERT Representationscited by 11
- 2021Patterns of Lexical Ambiguity in Contextualised Language Modelscited by 4
- 2020A Practical Guide to Studying Emergent Communication through Grounded Language Gamescited by 3
- 2022Abstraction not Memory: BERT and the English Article Systemcited by 1
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.