mlsanityData to 7 Oct 2026

Discovering Universal Geometry in Embeddings with ICA

arXiv 2305.13175 ↗First posted 22 May 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

This study utilizes Independent Component Analysis (ICA) to unveil a consistent semantic structure within embeddings of words or images. Our approach extracts independent semantic components from the embeddings of a pre-trained model by leveraging anisotropic information that remains after the whitening process in Principal Component Analysis (PCA). We demonstrate that each embedding can be expressed as a composition of a few intrinsic interpretable axes and that these semantic axes remain consistent across different languages, algorithms, and modalities. The discovery of a universal semantic structure in the geometric patterns of embeddings enhances our understanding of the representations in embeddings.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

34 of 68 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  10. 2019How multilingual is Multilingual BERT?cited by 642
  11. 2017Word Translation Without Parallel Datacited by 480
  12. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
22 more
  1. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  2. 2018Learning Word Vectors for 157 Languagescited by 279
  3. 2013k-Sparse Autoencoderscited by 209
  4. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  5. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  6. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  7. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  8. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  9. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  10. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  11. 2018Gromov-Wasserstein Alignment of Word Embedding Spacescited by 119
  12. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  13. 2018Unsupervised Alignment of Embeddings with Wasserstein Procrustescited by 96
  14. 2022The Geometry of Multilingual Language Model Representationscited by 94
  15. 2015Optimal whitening and decorrelationcited by 93
  16. 2017SPINE: SParse Interpretable Neural Embeddingscited by 55
  17. 2016SimVerb-3500: A Large-Scale Evaluation Set of Verb Similaritycited by 44
  18. 2015Sparse Overcomplete Word Vector Representationscited by 43
  19. 2020The POLAR Framework: Polar Opposites Enable Interpretability of Pre-Trained Word Embeddingscited by 15
  20. 2023SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddingscited by 9
  21. 2022Quantized Wasserstein Procrustes Alignment of Word Embedding Spacescited by 6
  22. 2018RegNet: Learning the Optimization of Direct Image-to-Image Pose Registrationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 5; 2026: 3.

19 papers.

Sort:

  1. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  2. 2024Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddingscited by 6
  3. 2024A Semantic Distance Metric Learning approach for Lexical Semantic Change Detectioncited by 5
  4. 2024Revisiting Cosine Similarity via Normalized ICA-transformed Embeddingscited by 5
  5. 2024Investigating the Contextualised Word Embedding Dimensions Specified for Contextual and Temporal Semantic Changescited by 4
  6. 2025SCDTour: Embedding Axis Ordering and Merging for Interpretable Semantic Change Detectioncited by 4
  7. 2025Hierarchical Semantic Retrieval with Cobwebcited by 4
  8. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3
  9. 2025mini-vec2vec: Scaling Universal Geometry Alignment with Linear Transformationscited by 3
  10. 2024Exploring Intra and Inter-language Consistency in Embeddings with ICAcited by 2
  11. 2024Fine-grained length controllable video captioning with ordinal embeddingscited by 2
  12. 2024Norm of Mean Contextualized Embeddings Determines their Variancecited by 1
7 more
  1. 2024Understanding Higher-Order Correlations Among Semantic Components in Embeddingscited by 1
  2. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionarycited by 1
  3. 2024DefSent+: Improving sentence embeddings of language models by projecting definition sentences into a quasi-isotropic or isotropic vector space of unlimited dictionary entriescited by 0
  4. 2024S3 -- Semantic Signal Separationcited by 0
  5. 2024Transforming Hidden States into Binary Semantic Featurescited by 0
  6. 2026Emergent Symbolic Structure in Health Foundation Models: Extraction, Alignment, and Cross-Modal Transfercited by 0
  7. 2026Structure Retention in Embedding Spaces as a Predictor of Benchmark Performancecited by 0