mlsanityData to 7 Oct 2026

On the proper role of linguistically-oriented deep net analysis in linguistic theorizing

arXiv 2106.08694 ↗First posted 16 Jun 2021cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

A lively research field has recently emerged that uses experimental methods to probe the linguistic behavior of modern deep networks. While work in this tradition often reports intriguing results about the grammatical skills of deep nets, it is not clear what their implications for linguistic theorizing should be. As a consequence, linguistically-oriented deep net analysis has had very little impact on linguistics at large. In this chapter, I suggest that deep networks should be treated as theories making explicit predictions about the acceptability of linguistic utterances. I argue that, if we overcome some obstacles standing in the way of seriously pursuing this idea, we will gain a powerful new theoretical tool, complementary to mainstream algebraic approaches.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 26 of its 69 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

26 of 69 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  12. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
14 more
  1. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  2. 2019Assessing BERT's Syntactic Abilitiescited by 191
  3. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  4. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  5. 2020Explainable Deep Learning: A Field Guide for the Uninitiatedcited by 127
  6. 2020Syntactic Structure from Deep Learningcited by 99
  7. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  8. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  9. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  10. 2019Distributional Semantics and Linguistic Theorycited by 53
  11. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  12. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  13. 2018RNNs Implicitly Implement Tensor Product Representationscited by 25
  14. 2020What they do when in doubt: a study of inductive biases in seq2seq learnerscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 9; 2024: 13; 2025: 9; 2026: 2.

43 papers.

Sort:

By area:

  1. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  2. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  3. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  4. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  5. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  6. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  7. 2023Dissociating language and thought in large language modelscited by 27
  8. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  9. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  10. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  11. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  12. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
31 more
  1. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  2. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  3. 2024Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrationscited by 12
  4. 2024Scope Ambiguities in Large Language Modelscited by 11
  5. 2022Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer?cited by 10
  6. 2023Why Linguistics Will Thrive in the 21st Century: A Reply to Piantadosi (2023)cited by 9
  7. 2023Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanitiescited by 9
  8. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  9. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  10. 2021Schr"odinger's Tree -- On Syntax and Neural Language Modelscited by 6
  11. 2023Learning the meanings of function words from grounded language using a visual question answering modelcited by 6
  12. 2024Transformers, Contextualism, and Polysemycited by 5
  13. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  14. 2024Computational Models to Study Language Processing in the Human Brain: A Surveycited by 4
  15. 2024Interpretability of Language Models via Task Spacescited by 4
  16. 2025Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Modelscited by 4
  17. 2021How Does Adversarial Fine-Tuning Benefit BERT?cited by 3
  18. 2025Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossiblecited by 3
  19. 2024Modelling Language using Large Language Modelscited by 2
  20. 2023Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Wayscited by 1
  21. 2024No Such Thing as a General Learner: Language models and their dual optimizationcited by 1
  22. 2025Language Models Largely Exhibit Human-like Constituent Ordering Preferencescited by 1
  23. 2021Transformers in the loop: Polarity in neural models of languagecited by 0
  24. 2022Integrating Linguistic Theory and Neural Language Modelscited by 0
  25. 2024Black Big Boxes: Tracing Adjective Order Preferences in Large Language Modelscited by 0
  26. 2024On the Compatibility of Generative AI and Generative Linguisticscited by 0
  27. 2025Generative Linguistics, Large Language Models, and the Social Nature of Scientific Successcited by 0
  28. 2025A conclusive remark on linguistic theorizing and language modelingcited by 0
  29. 2025Pragmatics beyond humans: meaning, communication, and LLMscited by 0
  30. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Modelscited by 0
  31. 2026A Formal Limitation on Learning Human Language From Textual Corporacited by 0