mlsanityData to 7 Oct 2026

Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations

arXiv 2109.14989 ↗First posted 30 Sep 2021cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

We investigate the extent to which modern, neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors which interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalisations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model's internal states.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 40 of its 103 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

40 of 103 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
28 more
  1. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  3. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  4. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  5. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2016Exploring the Limits of Language Modelingcited by 172
  8. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  9. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  10. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  11. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  12. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  13. 2020UnNatural Language Inferencecited by 68
  14. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  15. 2019Distributional Semantics and Linguistic Theorycited by 53
  16. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  17. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  18. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  19. 2020CxGBERT: BERT meets Construction Grammarcited by 32
  20. 2021A Non-Linear Structural Probecited by 31
  21. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  22. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  23. 2019Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshopcited by 22
  24. 2018A Neural Model of Adaptation in Readingcited by 18
  25. 2020Discourse structure interacts with reference but not syntax in neural language modelscited by 17
  26. 2021On the Limits of Minimal Pairs in Contrastive Evaluationcited by 15
  27. 2020diagNNose: A Library for Neural Activation Analysiscited by 7
  28. 2020Overestimation of Syntactic Representationin Neural Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 9; 2025: 5; 2026: 11.

44 papers.

Sort:

By area:

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 176
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2022Do Large Language Models know what humans know?cited by 60
  5. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  6. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  7. 2023Do large language models resemble humans in language use?cited by 34
  8. 2024The broader spectrum of in-context learningcited by 33
  9. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  10. 2022Language model acceptability judgements are not always robust to contextcited by 22
  11. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  12. 2024Do Language Models Exhibit Human-like Structural Priming Effects?cited by 21
32 more
  1. 2023Machine Psychologycited by 16
  2. 2023Evaluating Neural Language Models as Cognitive Models of Language Acquisitioncited by 15
  3. 2025Herd Behavior: Investigating Peer Influence in LLM-based Multi-Agent Systemscited by 15
  4. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  5. 2023Using Artificial Populations to Study Psychological Phenomena in Neural Modelscited by 8
  6. 2024Large Language Model Recall Uncertainty is Modulated by the Fan Effectcited by 7
  7. 2025Do language models accommodate their users? A study of linguistic convergencecited by 7
  8. 2023Attribution and Alignment: Effects of Local Context Repetition on Utterance Production and Comprehension in Dialoguecited by 4
  9. 2024Interpretability of Language Models via Task Spacescited by 4
  10. 2023Large Language Models Are Partially Primed in Pronoun Interpretationcited by 3
  11. 2024Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistentlycited by 3
  12. 2024SPAWNing Structural Priming Predictions from a Cognitively Motivated Parsercited by 3
  13. 2024Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Primingcited by 3
  14. 2024Is In-Context Learning a Type of Error-Driven Learning? Evidence from the Inverse Frequency Effect in Structural Primingcited by 3
  15. 2023Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativizationcited by 2
  16. 2026LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathologycited by 2
  17. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  18. 2026In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Headscited by 1
  19. 2026Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialoguecited by 1
  20. 2026Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedbackcited by 1
  21. 2023Multimodal Shannon Game with Imagescited by 0
  22. 2023Crosslingual Structural Priming and the Pre-Training Dynamics of Bilingual Language Modelscited by 0
  23. 2024Animate, or Inanimate, That is the Question for Large Language Modelscited by 0
  24. 2025Towards Human Cognition: Visual Context Guides Syntactic Priming in Fusion-Encoded Modelscited by 0
  25. 2025Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and Englishcited by 0
  26. 2025Unsupervised Classification of English Words Based on Phonological Information: Discovery of Germanic and Latinate Clusterscited by 0
  27. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  28. 2026Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learningcited by 0
  29. 2026Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Docited by 0
  30. 2026Attention is Case-Sensitivecited by 0
  31. 2026Automatic or Controlled? Repetition Priming Reveals Divergent Processing in Base LLMs, Instruct LLMs, and Humanscited by 0
  32. 2026Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generationcited by 0