# Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming

arXiv: https://arxiv.org/abs/2010.03010 · HTML: https://mlsanity.com/paper/2010.03010/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 41)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Do language models learn typicality judgments from text?](https://mlsanity.com/paper/2105.02987/) (2021)
- [Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models](https://mlsanity.com/paper/2310.16570/) (2023)
- [Sorting through the noise: Testing robustness of information processing in pre-trained language models](https://mlsanity.com/paper/2109.12393/) (2021)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [The Curious Case of Absolute Position Embeddings](https://mlsanity.com/paper/2210.12574/) (2022)
- [COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models](https://mlsanity.com/paper/2210.01963/) (2022)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?](https://mlsanity.com/paper/2107.09648/) (2021)
- [Do PLMs Know and Understand Ontological Knowledge?](https://mlsanity.com/paper/2309.05936/) (2023)
- [Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale](https://mlsanity.com/paper/2510.24963/) (2025)
- [Lexical Generalization Improves with Larger Models and Longer Training](https://mlsanity.com/paper/2210.12673/) (2022)
- [Can Peanuts Fall in Love with Distributional Semantics?](https://mlsanity.com/paper/2301.08731/) (2023)
- [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024)
- [John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs](https://mlsanity.com/paper/2106.01060/) (2021)
- [Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning](https://mlsanity.com/paper/2106.01207/) (2021)
- [Can Pretrained Language Models (Yet) Reason Deductively?](https://mlsanity.com/paper/2210.06442/) (2022)
- [Collateral facilitation in humans and language models](https://mlsanity.com/paper/2211.05198/) (2022)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models](https://mlsanity.com/paper/2406.11097/) (2024)
- [Quantifying the Task-Specific Information in Text-Based Classifications](https://mlsanity.com/paper/2110.08931/) (2021)
- [Large Language Models Are Partially Primed in Pronoun Interpretation](https://mlsanity.com/paper/2305.16917/) (2023)
- [Logically Consistent Adversarial Attacks for Soft Theorem Provers](https://mlsanity.com/paper/2205.00047/) (2022)
- [Discourse Context Predictability Effects in Hindi Word Order](https://mlsanity.com/paper/2210.13940/) (2022)
- [Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain](https://mlsanity.com/paper/2304.14745/) (2023)
- [Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events](https://mlsanity.com/paper/2506.06808/) (2025)
- [Counteracts: Testing Stereotypical Representation in Pre-trained Language Models](https://mlsanity.com/paper/2301.04347/) (2023)
- [Characterizing and Measuring Linguistic Dataset Drift](https://mlsanity.com/paper/2305.17127/) (2023)

## Abstract (arXiv)

Models trained to estimate word probabilities in context have become ubiquitous in natural language processing. How do these models use lexical cues in context to inform their word probabilities? To answer this question, we present a case study analyzing the pre-trained BERT model with tests informed by semantic priming. Using English lexical stimuli that show priming in humans, we find that BERT too shows "priming," predicting a word with greater probability when the context includes a related word versus an unrelated one. This effect decreases as the amount of information provided by the context increases. Follow-up analysis shows BERT to be increasingly distracted by related prime words as context becomes more informative, assigning lower probabilities to related words. Our findings highlight the importance of considering contextual constraint effects when studying word prediction in these models, and highlight possible parallels with human processing.