# Discovering Differences in the Representation of People using Contextualized Semantic Axes

arXiv: https://arxiv.org/abs/2210.12170 · HTML: https://mlsanity.com/paper/2210.12170/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 94 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change](https://mlsanity.com/paper/1606.02821/) (2016)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models](https://mlsanity.com/paper/2110.00672/) (2021)
- [Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift](https://mlsanity.com/paper/1912.01072/) (2019)
- [Uncovering Implicit Gender Bias in Narratives through Commonsense Inference](https://mlsanity.com/paper/2109.06437/) (2021)
- [A Framework for the Computational Linguistic Analysis of Dehumanization](https://mlsanity.com/paper/2003.03014/) (2020)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment](https://mlsanity.com/paper/1806.05521/) (2018)
- [A Quantitative Approach to Understanding Online Antisemitism](https://mlsanity.com/paper/1809.01644/) (2018)
- [Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model](https://mlsanity.com/paper/2106.02596/) (2021)
- [FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding](https://mlsanity.com/paper/2002.08608/) (2020)
- [Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora](https://mlsanity.com/paper/2112.14330/) (2021)
- [Characterizing English Variation across Social Media Communities with BERT](https://mlsanity.com/paper/2102.06820/) (2021)
- [Frequency-based Distortions in Contextualized Word Embeddings](https://mlsanity.com/paper/2104.08465/) (2021)
- [Unsupervised Discovery of Gendered Language through Latent-Variable Modeling](https://mlsanity.com/paper/1906.04760/) (2019)
- [On the Robustness of Language Encoders against Grammatical Errors](https://mlsanity.com/paper/2005.05683/) (2020)
- [The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings](https://mlsanity.com/paper/2001.09876/) (2020)

## Built on by (most cited first, 12 of 12)

- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Unsupervised Contrast-Consistent Ranking with Language Models](https://mlsanity.com/paper/2309.06991/) (2023)
- [From tools to thieves: Measuring and understanding public perceptions of AI through crowdsourced metaphors](https://mlsanity.com/paper/2501.18045/) (2025)
- [Wikibio: a Semantic Resource for the Intersectional Analysis of Biographical Events](https://mlsanity.com/paper/2306.09505/) (2023)
- [Adjusting Interpretable Dimensions in Embedding Space with Human Judgments](https://mlsanity.com/paper/2404.02619/) (2024)
- [Rethinking Word Similarity: Semantic Similarity through Classification Confusion](https://mlsanity.com/paper/2502.05704/) (2025)
- [STEREODISCO: Discovering Stereotypicality in LLMs](https://mlsanity.com/paper/2607.27824/) (2026)
- [CIDER: Context sensitive sentiment analysis for short-form text](https://mlsanity.com/paper/2307.07864/) (2023)
- [Capturing Differences in Character Representations Between Communities: An Initial Study with Fandom](https://mlsanity.com/paper/2409.11170/) (2024)
- [LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioning](https://mlsanity.com/paper/2509.22211/) (2025)
- [Interpretable dimensions support an effect of agentivity and telicity on split intransitivity](https://mlsanity.com/paper/2511.16824/) (2025)
- [Language Models as Measurement Apparatus for Culture](https://mlsanity.com/paper/2607.02459/) (2026)

## Abstract (arXiv)

A common paradigm for identifying semantic differences across social and temporal contexts is the use of static word embeddings and their distances. In particular, past work has compared embeddings against "semantic axes" that represent two opposing concepts. We extend this paradigm to BERT embeddings, and construct contextualized axes that mitigate the pitfall where antonyms have neighboring representations. We validate and demonstrate these axes on two people-centric datasets: occupations from Wikipedia, and multi-platform discussions in extremist, men's communities over fourteen years. In both studies, contextualized semantic axes can characterize differences among instances of the same word type. In the latter study, we show that references to women and the contexts around them have become more detestable over time.