# Addressing Token Uniformity in Transformers via Singular Value Transformation

arXiv: https://arxiv.org/abs/2208.11790 · HTML: https://mlsanity.com/paper/2208.11790/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models](https://mlsanity.com/paper/2002.06652/) (2020)
- [Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities](https://mlsanity.com/paper/1902.08077/) (2019)
- [IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization](https://mlsanity.com/paper/2005.02178/) (2020)
- [Staying in Shape: Learning Invariant Shape Representations using Contrastive Learning](https://mlsanity.com/paper/2107.03552/) (2021)
- [A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks](https://mlsanity.com/paper/1602.01321/) (2016)

## Built on by (most cited first, 14 of 14)

- [ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond](https://mlsanity.com/paper/2303.06562/) (2023)
- [Setting the Record Straight on Transformer Oversmoothing](https://mlsanity.com/paper/2401.04301/) (2024)
- [Graph Convolutions Enrich the Self-Attention in Transformers!](https://mlsanity.com/paper/2312.04234/) (2023)
- [Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective](https://mlsanity.com/paper/2406.17969/) (2024)
- [Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recovery](https://mlsanity.com/paper/2309.01943/) (2023)
- [Generalized Probabilistic Attention Mechanism in Transformers](https://mlsanity.com/paper/2410.15578/) (2024)
- [The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA](https://mlsanity.com/paper/2609.03090/) (2026)
- [Sequential Recommendation with Controllable Diversification: Representation Degeneration and Diversity](https://mlsanity.com/paper/2306.11986/) (2023)
- [Why "classic" Transformers are shallow and how to make them go deep](https://mlsanity.com/paper/2312.06182/) (2023)
- [On the Role of Hidden States of Modern Hopfield Network in Transformer](https://mlsanity.com/paper/2511.20698/) (2025)
- [Distinguishability Calibration to In-Context Learning](https://mlsanity.com/paper/2302.06198/) (2023)
- [Unsupervised Speech Representation Pooling Using Vector Quantization](https://mlsanity.com/paper/2304.03940/) (2023)
- [BioNCERE: Non-Contrastive Enhancement For Relation Extraction In Biomedical Texts](https://mlsanity.com/paper/2410.23583/) (2024)
- [Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure](https://mlsanity.com/paper/2608.09417/) (2026)

## Abstract (arXiv)

Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the `token uniformity' problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks. Our source code is available at https://github.com/hanqi-qi/tokenUni.git.