# SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings

arXiv: https://arxiv.org/abs/2301.04704 · HTML: https://mlsanity.com/paper/2301.04704/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 11 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [Sparse Overcomplete Word Vector Representations](https://mlsanity.com/paper/1506.02004/) (2015)
- [Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples](https://mlsanity.com/paper/1912.04853/) (2019)
- [SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment](https://mlsanity.com/paper/1806.05521/) (2018)
- [FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding](https://mlsanity.com/paper/2002.08608/) (2020)
- [Quantifying the Contextualization of Word Representations with Semantic Class Probing](https://mlsanity.com/paper/2004.12198/) (2020)
- [The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings](https://mlsanity.com/paper/2001.09876/) (2020)
- [Visually Analyzing Contextualized Embeddings](https://mlsanity.com/paper/2009.02554/) (2020)
- [BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward](https://mlsanity.com/paper/2003.02738/) (2020)

## Built on by (most cited first, 9 of 9)

- [When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers](https://mlsanity.com/paper/2504.10957/) (2025)
- [Discovering Universal Geometry in Embeddings with ICA](https://mlsanity.com/paper/2305.13175/) (2023)
- [Unsupervised Contrast-Consistent Ranking with Language Models](https://mlsanity.com/paper/2309.06991/) (2023)
- [Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings](https://mlsanity.com/paper/2401.06112/) (2024)
- [Profiling Bias in LLMs: Stereotype Dimensions in Contextual Word Embeddings](https://mlsanity.com/paper/2411.16527/) (2024)
- [STEREODISCO: Discovering Stereotypicality in LLMs](https://mlsanity.com/paper/2607.27824/) (2026)
- [Interpretable Syntactic Representations Enable Hierarchical Word Vectors](https://mlsanity.com/paper/2411.08384/) (2024)
- [Visualising Information Flow in Word Embeddings with Diffusion Tensor Imaging](https://mlsanity.com/paper/2601.05713/) (2026)
- [Linguistic Features for Interpretable Textual Entailment](https://mlsanity.com/paper/2609.24932/) (2026)

## Abstract (arXiv)

Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored thepotential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.