# Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings

arXiv: https://arxiv.org/abs/2508.13729 · HTML: https://mlsanity.com/paper/2508.13729/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Are Emergent Abilities in Large Language Models just In-Context Learning?](https://mlsanity.com/paper/2309.01809/) (2023)
- [CogniVal: A Framework for Cognitive Word Embedding Evaluation](https://mlsanity.com/paper/1909.09001/) (2019)
- [Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge](https://mlsanity.com/paper/2010.13057/) (2020)
- [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023)
- [Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity](https://mlsanity.com/paper/1711.09285/) (2017)
- [Detecting Conceptual Abstraction in LLMs](https://mlsanity.com/paper/2404.15848/) (2024)

## Built on by (most cited first, 2 of 2)

- [From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery](https://mlsanity.com/paper/2607.12474/) (2026)
- [LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models](https://mlsanity.com/paper/2601.22928/) (2026)

## Abstract (arXiv)

Understanding what knowledge is implicitly encoded in deep learning models is essential for improving the interpretability of AI systems. This paper examines common methods to explain the knowledge encoded in word embeddings, which are core elements of large language models (LLMs). These methods typically involve mapping embeddings onto collections of human-interpretable semantic features, known as feature norms. Prior work assumes that accurately predicting these semantic features from the word embeddings implies that the embeddings contain the corresponding knowledge. We challenge this assumption by demonstrating that prediction accuracy alone does not reliably indicate genuine feature-based interpretability. We show that these methods can successfully predict even random information, concluding that the results are predominantly determined by an algorithmic upper bound rather than meaningful semantic representation in the word embeddings. Consequently, comparisons between datasets based solely on prediction performance do not reliably indicate which dataset is better captured by the word embeddings. Our analysis illustrates that such mappings primarily reflect geometric similarity within vector spaces rather than indicating the genuine emergence of semantic properties.