# Vector Arithmetic in Concept and Token Subspaces

arXiv: https://arxiv.org/abs/2511.18162 · HTML: https://mlsanity.com/paper/2511.18162/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Builds on (most cited first, 5 of 5)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [The Dual-Route Model of Induction](https://mlsanity.com/paper/2504.03022/) (2025)

## Built on by (most cited first, 4 of 4)

- [Transformers learn factored representations](https://mlsanity.com/paper/2602.02385/) (2026)
- [The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research](https://mlsanity.com/paper/2602.18458/) (2026)
- [Using OCR Heads to Verbalize Image Semantics](https://mlsanity.com/paper/2609.18823/) (2026)
- [LLMs program using Reusable and Composable Abstract Representations](https://mlsanity.com/paper/2610.19238/) (2026)

## Abstract (arXiv)

In order to predict the next token, LLMs must represent semantic and surface-level information about the current word. Previous work identified two types of attention heads that disentangle this information: (i) Concept induction heads, which copy word meanings, and (ii) Token induction heads, which copy literal token representations (Feucht et al., 2025). We show that these heads can be used to identify subspaces of model activations that exhibit coherent semantic structure in Llama-2-7b. Specifically, when we transform hidden states using the attention weights of concept heads, we are able to more accurately perform parallelogram arithmetic (Mikolov et al., 2013) on the resulting hidden states, e.g., showing that "Athens" - "Greece" + "China" = "Beijing". This transformation allows for much higher nearest-neighbor accuracy (80%) than direct use of raw hidden states (47%). Analogously, we show that token heads allow for transformations that reveal surface-level word information in hidden states, allowing for operations like "coding" - "code" + "dance" = "dancing".