mlsanityData to 7 Oct 2026

Reconstruction Probing

arXiv 2212.10792 ↗First posted 21 Dec 2022cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose reconstruction probing, a new analysis method for contextualized representations based on reconstruction probabilities in masked language models (MLMs). This method relies on comparing the reconstruction probabilities of tokens in a given sequence when conditioned on the representation of a single token that has been fully contextualized and when conditioned on only the decontextualized lexical prior of the model. This comparison can be understood as quantifying the contribution of contextualization towards reconstruction -- the difference in the reconstruction probabilities can only be attributed to the representational change of the single token induced by contextualization. We apply this analysis to three MLMs and find that contextualization boosts reconstructability of tokens that are close to the token being reconstructed in terms of linear and syntactic distance. Furthermore, we extend our analysis to finer-grained decomposition of contextualized representations, and we find that these boosts are largely attributable to static and positional embeddings at the input layer.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

22 of 39 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2019Assessing BERT's Syntactic Abilitiescited by 191
  12. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
10 more
  1. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  2. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  3. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  4. 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
  5. 2020Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding wordscited by 28
  6. 2020Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysiscited by 18
  7. 2021Probing for Bridging Inference in Transformer Language Modelscited by 14
  8. 2022Probing for Constituency Structure in Neural Language Modelscited by 13
  9. 2021Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowordscited by 9
  10. 2021Effect of Post-processing on Contextualized Word Representationscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  2. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1