# Transforming Hidden States into Binary Semantic Features

arXiv: https://arxiv.org/abs/2409.19813 · HTML: https://mlsanity.com/paper/2409.19813/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Builds on (most cited first, 6 of 6)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Discovering Universal Geometry in Embeddings with ICA](https://mlsanity.com/paper/2305.13175/) (2023)
- [Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings](https://mlsanity.com/paper/2401.06112/) (2024)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Exploring Intra and Inter-language Consistency in Embeddings with ICA](https://mlsanity.com/paper/2406.12474/) (2024)

## Abstract (arXiv)

Large language models follow a lineage of many NLP applications that were directly inspired by distributional semantics, but do not seem to be closely related to it anymore. In this paper, we propose to employ the distributional theory of meaning once again. Using Independent Component Analysis to overcome some of its challenging aspects, we show that large language models represent semantic features in their hidden states.