# Probing for Bridging Inference in Transformer Language Models

arXiv: https://arxiv.org/abs/2104.09400 · HTML: https://mlsanity.com/paper/2104.09400/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Do Attention Heads in BERT Track Syntactic Dependencies?](https://mlsanity.com/paper/1911.12246/) (2019)
- [Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations](https://mlsanity.com/paper/1910.01157/) (2019)
- [Bridging Anaphora Resolution as Question Answering](https://mlsanity.com/paper/2004.07898/) (2020)
- [Enhanced Word Representations for Bridging Anaphora Resolution](https://mlsanity.com/paper/1803.04790/) (2018)
- [A Deterministic Algorithm for Bridging Anaphora Resolution](https://mlsanity.com/paper/1811.05721/) (2018)
- [Fine-grained Information Status Classification Using Discourse Context-Aware BERT](https://mlsanity.com/paper/2010.14759/) (2020)

## Built on by (most cited first, 14 of 14)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Holmes: A Benchmark to Assess the Linguistic Competence of Language Models](https://mlsanity.com/paper/2404.18923/) (2024)
- [How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression](https://mlsanity.com/paper/2408.04532/) (2024)
- [Controlled Text Reduction](https://mlsanity.com/paper/2210.13449/) (2022)
- [Text-based NP Enrichment](https://mlsanity.com/paper/2109.12085/) (2021)
- [SocioProbe: What, When, and Where Language Models Learn about Sociodemographics](https://mlsanity.com/paper/2211.04281/) (2022)
- [Debiasing should be Good and Bad: Measuring the Consistency of Debiasing Techniques in Language Models](https://mlsanity.com/paper/2305.14307/) (2023)
- [What can Neural Referential Form Selectors Learn?](https://mlsanity.com/paper/2108.06806/) (2021)
- [Probing Script Knowledge from Pre-Trained Models](https://mlsanity.com/paper/2204.10176/) (2022)
- [On the Robustness of Transformers against Context Hijacking for Linear Classification](https://mlsanity.com/paper/2502.15609/) (2025)
- [Reconstruction Probing](https://mlsanity.com/paper/2212.10792/) (2022)
- [Automatic Generation of Inference Making Questions for Reading Comprehension Assessments](https://mlsanity.com/paper/2506.08260/) (2025)
- [Human-Like Anaphor Resolution in Large Language Models](https://mlsanity.com/paper/2608.05630/) (2026)
- [Rate-Coding Bundle Memory: A Unified Model of Memory and Control for Symbolic Computation in the Brain](https://mlsanity.com/paper/2608.29189/) (2026)

## Abstract (arXiv)

We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to language models play an important role in the inference.