mlsanityData to 7 Oct 2026

Exploring the Role of BERT Token Representations to Explain Sentence Probing Results

arXiv 2104.01477 ↗First posted 3 Apr 2021cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Several studies have been carried out on revealing linguistic features captured by BERT. This is usually achieved by training a diagnostic classifier on the representations obtained from different layers of BERT. The subsequent classification accuracy is then interpreted as the ability of the model in encoding the corresponding linguistic property. Despite providing insights, these studies have left out the potential role of token representations. In this paper, we provide a more in-depth analysis on the representation space of BERT in search for distinct and meaningful subspaces that can explain the reasons behind these probing results. Based on a set of probing tasks and with the help of attribution methods we show that BERT tends to encode meaningful knowledge in specific token representations (which are often ignored in standard classification setups), allowing the model to detect syntactic and semantic abnormalities, and to distinctively separate grammatical number and tense subspaces.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 43 of its 49 references that are in the map. First posted on arXiv 3 Apr 2021.

Builds on

43 of 49 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2018Neural Network Acceptability Judgmentscited by 952
31 more
  1. 2019Attention is not Explanationcited by 916
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2019How multilingual is Multilingual BERT?cited by 642
  4. 2019Designing and Interpreting Probes with Control Taskscited by 608
  5. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  6. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  7. 2019Is Attention Interpretable?cited by 387
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  9. 2019Revealing the Dark Secrets of BERTcited by 300
  10. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  11. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  13. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  14. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  15. 2019Assessing BERT's Syntactic Abilitiescited by 191
  16. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  17. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  18. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  19. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  20. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  21. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  22. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  23. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  24. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  25. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  26. 2019AllenNLP Interpret: A Framework for Explaining Predictions of NLP Modelscited by 74
  27. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  28. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  29. 2020Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding wordscited by 28
  30. 2021On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classificationcited by 21
  31. 2020An information theoretic view on selecting linguistic probescited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 2; 2024: 10; 2025: 10; 2026: 2.

29 papers.

Sort:

By area:

  1. 2023Explainability for Large Language Models: A Surveycited by 331
  2. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  3. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  4. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  5. 2022Joint processing of linguistic properties in brains and language modelscited by 37
  6. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  7. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  8. 2022AdapLeR: Speeding up Inference by Adaptive Length Reductioncited by 19
  9. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
  10. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  11. 2024Explaining Text Similarity in Transformer Modelscited by 9
  12. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
17 more
  1. 2025Towards Transparent AI: A Survey on Explainable Large Language Modelscited by 7
  2. 2025SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Modelscited by 6
  3. 2025Saliency-driven Dynamic Token Pruning for Large Language Modelscited by 5
  4. 2022Life after BERT: What do Other Muppets Understand about Language?cited by 4
  5. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  6. 2025WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learningcited by 3
  7. 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
  8. 2024IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?cited by 2
  9. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  10. 2024A Theoretical Survey on Foundation Modelscited by 1
  11. 2025On the reliability of feature attribution methods for speech classificationcited by 1
  12. 2025How do Transformer Embeddings Represent Compositions? A Functional Analysiscited by 1
  13. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  14. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  15. 2026DRIV-EX: Counterfactual Explanations for Driving LLMscited by 1
  16. 2024VI-OOD: A Unified Representation Learning Framework for Textual Out-of-distribution Detectioncited by 0
  17. 2024Automated Trustworthiness Oracle Generation for Machine Learning Text Classifierscited by 0