mlsanityData to 7 Oct 2026

Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings

arXiv 2401.06112 ↗First posted 11 Jan 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Word embedding is one of the most important components in natural language processing, but interpreting high-dimensional embeddings remains a challenging problem. To address this problem, Independent Component Analysis (ICA) is identified as an effective solution. ICA-transformed word embeddings reveal interpretable semantic axes; however, the order of these axes are arbitrary. In this study, we focus on this property and propose a novel method, Axis Tour, which optimizes the order of the axes. Inspired by Word Tour, a one-dimensional word embedding method, we aim to improve the clarity of the word embedding space by maximizing the semantic continuity of the axes. Furthermore, we show through experiments on downstream tasks that Axis Tour yields better or comparable low-dimensional embeddings compared to both PCA and ICA.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 12 of its 42 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

12 of 42 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  5. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  6. 2018SemAxis: A Lightweight Framework to Characterize Domain-Specific Word Semantics Beyond Sentimentcited by 25
  7. 2020FrameAxis: Characterizing Microframe Bias and Intensity with Word Embeddingcited by 21
  8. 2023Discovering Universal Geometry in Embeddings with ICAcited by 19
  9. 2020The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddingscited by 15
  10. 2017How to evaluate word embeddings? On importance of data efficiency and simple supervised taskscited by 12
  11. 2023SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddingscited by 9
  12. 2019Examining Structure of Word Embeddings with PCAcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 3.

6 papers.

  1. 2024Revisiting Cosine Similarity via Normalized ICA-transformed Embeddingscited by 5
  2. 2026Measuring Affinity between Attention-Head Weight Subspaces via the Projection Kernelcited by 5
  3. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  4. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionarycited by 1
  5. 2024Transforming Hidden States into Binary Semantic Featurescited by 0
  6. 2026Structure Retention in Embedding Spaces as a Predictor of Benchmark Performancecited by 0