mlsanityData to 14 Oct 2026

Addressing Token Uniformity in Transformers via Singular Value Transformation

arXiv 2208.11790 ↗First posted 24 Aug 2022cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the `token uniformity' problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks. Our source code is available at https://github.com/hanqi-qi/tokenUni.git.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 23 of its 41 references that are in the map. First posted on arXiv 24 Aug 2022.

Builds on

23 of 41 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Layer Normalizationcited by 5,669
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2018Group Normalizationcited by 1,524
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2018Neural Network Acceptability Judgmentscited by 958
11 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  2. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  3. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  4. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  5. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  6. 2021WhiteningBERT: An Easy Unsupervised Sentence Embedding Approachcited by 69
  7. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
  8. 2019Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearitiescited by 14
  9. 2020IsoBN: Fine-Tuning BERT with Isotropic Batch Normalizationcited by 12
  10. 2021Staying in Shape: Learning Invariant Shape Representations using Contrastive Learningcited by 4
  11. 2016A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 4; 2025: 1; 2026: 2.

14 papers.

Sort:

By area:

  1. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  2. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  3. 2023Graph Convolutions Enrich the Self-Attention in Transformers!cited by 11
  4. 2024Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspectivecited by 10
  5. 2023Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recoverycited by 7
  6. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
  7. 2026The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQAcited by 2
  8. 2023Sequential Recommendation with Controllable Diversification: Representation Degeneration and Diversitycited by 1
  9. 2023Why "classic" Transformers are shallow and how to make them go deepcited by 1
  10. 2025On the Role of Hidden States of Modern Hopfield Network in Transformercited by 1
  11. 2023Distinguishability Calibration to In-Context Learningcited by 0
  12. 2023Unsupervised Speech Representation Pooling Using Vector Quantizationcited by 0
2 more
  1. 2024BioNCERE: Non-Contrastive Enhancement For Relation Extraction In Biomedical Textscited by 0
  2. 2026Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failurecited by 0