# Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings

arXiv: https://arxiv.org/abs/2401.06112 · HTML: https://mlsanity.com/paper/2401.06112/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentiment](https://mlsanity.com/paper/1806.05521/) (2018)
- [FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding](https://mlsanity.com/paper/2002.08608/) (2020)
- [Discovering Universal Geometry in Embeddings with ICA](https://mlsanity.com/paper/2305.13175/) (2023)
- [The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddings](https://mlsanity.com/paper/2001.09876/) (2020)
- [How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks](https://mlsanity.com/paper/1702.02170/) (2017)
- [SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings](https://mlsanity.com/paper/2301.04704/) (2023)
- [Examining Structure of Word Embeddings with PCA](https://mlsanity.com/paper/1906.00114/) (2019)

## Built on by (most cited first, 6 of 6)

- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernel](https://mlsanity.com/paper/2601.10266/) (2026)
- [SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detection](https://mlsanity.com/paper/2509.11818/) (2025)
- [ICA Lens: Interpreting Language Models Without Training Another Dictionary](https://mlsanity.com/paper/2606.11722/) (2026)
- [Transforming Hidden States into Binary Semantic Features](https://mlsanity.com/paper/2409.19813/) (2024)
- [Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance](https://mlsanity.com/paper/2605.22202/) (2026)

## Abstract (arXiv)

Word embedding is one of the most important components in natural language processing, but interpreting high-dimensional embeddings remains a challenging problem. To address this problem, Independent Component Analysis (ICA) is identified as an effective solution. ICA-transformed word embeddings reveal interpretable semantic axes; however, the order of these axes are arbitrary. In this study, we focus on this property and propose a novel method, Axis Tour, which optimizes the order of the axes. Inspired by Word Tour, a one-dimensional word embedding method, we aim to improve the clarity of the word embedding space by maximizing the semantic continuity of the axes. Furthermore, we show through experiments on downstream tasks that Axis Tour yields better or comparable low-dimensional embeddings compared to both PCA and ICA.