mlsanityData to 7 Oct 2026

Transformers in the loop: Polarity in neural models of language

arXiv 2109.03926 ↗First posted 8 Sep 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Representation of linguistic phenomena in computational language models is typically assessed against the predictions of existing linguistic theories of these phenomena. Using the notion of polarity as a case study, we show that this is not always the most adequate set-up. We probe polarity via so-called 'negative polarity items' (in particular, English 'any') in two pre-trained Transformer-based models (BERT and GPT-2). We show that - at least for polarity - metrics derived from language models are more consistent with data from psycholinguistic experiments than linguistic theory predictions. Establishing this allows us to more adequately evaluate the performance of language models and also to use language models to discover new insights into natural language grammar beyond existing linguistic theories. This work contributes to establishing closer ties between psycholinguistic experiments and experiments with language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 8 Sep 2021.

Builds on

22 of 37 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2021Causal Abstractions of Neural Networkscited by 401
  4. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  5. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  6. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  7. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  8. 2019Assessing BERT's Syntactic Abilitiescited by 191
  9. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  10. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  11. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  12. 2020Syntactic Structure from Deep Learningcited by 99
10 more
  1. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  2. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  3. 2019Can neural networks understand monotonicity reasoning?cited by 45
  4. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  5. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  6. 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
  7. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  8. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  9. 2020Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledgecited by 18
  10. 2021Language Modelling as a Multi-Task Problemcited by 10

Built on by

Not cited yet.