mlsanityData to 7 Oct 2026

The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models

arXiv 2311.05928 ↗First posted 10 Nov 2023cs.CL cs.AI cs.ITcited by 37

Not on the map. Why?

Abstract (arXiv)

In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

15 of 29 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  9. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  10. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  11. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  12. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
3 more
  1. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  2. 2018Persistent Homology and the Upper Box Dimensioncited by 15
  3. 2022The Architectural Bottleneck Principlecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 18.

37 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 48
  3. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  4. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  5. 2025Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMscited by 26
  6. 2024Your Transformer is Secretly Linearcited by 23
  7. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  8. 2025Large Language Models Encode Semantics and Alignment in Linearly Separable Representationscited by 14
  9. 2025Randomly Removing 50% of Dimensions in Text Embeddings has Minimal Impact on Retrieval and Classification Taskscited by 7
  10. 2025Layer-wise dynamic rank for compressing large language modelscited by 6
  11. 2024Whitening Consistently Improves Self-Supervised Learningcited by 5
  12. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
25 more
  1. 2025Geometric Metrics and LLMs: What They Measure and When They Workcited by 4
  2. 2025From Topology to Retrieval: Decoding Embedding Spaces with Unified Signaturescited by 4
  3. 2025Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimationcited by 3
  4. 2025Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insightscited by 3
  5. 2025TRACE for Tracking the Emergence of Semantic Representations in Transformerscited by 3
  6. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  7. 2026Uncovering the Latent Potential of Deep Intermediate Representationscited by 3
  8. 2025CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functionscited by 2
  9. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  10. 2024Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrievalcited by 1
  11. 2025Unveiling Intrinsic Dimension of Texts: from Academic Abstract to Creative Storycited by 1
  12. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  13. 2026Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Modelscited by 1
  14. 2026A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalizationcited by 1
  15. 2026When Do Concepts Become Functionally Sufficient During Language-Model Training?cited by 1
  16. 2025Breaking the Curse of Dimensionality: On the Stability of Modern Vector Retrievalcited by 0
  17. 2026Emergent Causal-Geometric Dynamics Across Depth in Large Language Modelscited by 0
  18. 2026TextTeacher: What Can Language Teach About Images?cited by 0
  19. 2026What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?cited by 0
  20. 2026IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignmentcited by 0
  21. 2026What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Modelscited by 0
  22. 2026Linear Probing Provides Robust and Efficient Detection of Machine-Generated Textcited by 0
  23. 2026A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddingscited by 0
  24. 2026How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Modelscited by 0
  25. 2026SPICE: Simple Polysemantic Feature Interpretation via Clustering-based Explanationcited by 0