# Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings

arXiv: https://arxiv.org/abs/2012.15353 · HTML: https://mlsanity.com/paper/2012.15353/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023)
- [AGENTiGraph: A Multi-Agent Knowledge Graph Framework for Interactive, Domain-Specific LLM Chatbots](https://mlsanity.com/paper/2508.02999/) (2025)
- [AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data](https://mlsanity.com/paper/2410.11531/) (2024)
- [BLCU-ICALL at SemEval-2022 Task 1: Cross-Attention Multitasking Framework for Definition Modeling](https://mlsanity.com/paper/2204.07701/) (2022)
- [Consistency and Coherence from Points of Contextual Similarity](https://mlsanity.com/paper/2112.11638/) (2021)
- [Tracking Semantic Change in Slovene: A Novel Dataset and Optimal Transport-Based Distance](https://mlsanity.com/paper/2402.16596/) (2024)
- [Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science](https://mlsanity.com/paper/2503.16515/) (2025)
- [Contrast-CAT: Contrasting Activations for Enhanced Interpretability in Transformer-based Text Classifiers](https://mlsanity.com/paper/2507.21186/) (2025)
- [From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification](https://mlsanity.com/paper/2509.21341/) (2025)
- [Improving Interpretability of Lexical Semantic Change with Neurobiological Features](https://mlsanity.com/paper/2602.09760/) (2026)
- [Differentiating Approach and Avoidance from Traditional Notions of Sentiment in Economic Contexts](https://mlsanity.com/paper/2112.02607/) (2021)
- [Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users](https://mlsanity.com/paper/2206.12368/) (2022)

## Abstract (arXiv)

Models based on the transformer architecture, such as BERT, have marked a crucial step forward in the field of Natural Language Processing. Importantly, they allow the creation of word embeddings that capture important semantic information about words in context. However, as single entities, these embeddings are difficult to interpret and the models used to create them have been described as opaque. Binder and colleagues proposed an intuitive embedding space where each dimension is based on one of 65 core semantic features. Unfortunately, the space only exists for a small dataset of 535 words, limiting its uses. Previous work (Utsumi, 2018, 2020, Turton, Vinson & Smith, 2020) has shown that Binder features can be derived from static embeddings and successfully extrapolated to a large new vocabulary. Taking the next step, this paper demonstrates that Binder features can be derived from the BERT embedding space. This provides contextualised Binder embeddings, which can aid in understanding semantic differences between words in context. It additionally provides insights into how semantic features are represented across the different layers of the BERT model.