# Optimal whitening and decorrelation

arXiv: https://arxiv.org/abs/1512.00809 · HTML: https://mlsanity.com/paper/1512.00809/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2015.

## Built on by (most cited first, 30 of 97)

- [RanPAC: Random Projections and Pre-trained Models for Continual Learning](https://mlsanity.com/paper/2307.02251/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Generalized Shape Metrics on Neural Representations](https://mlsanity.com/paper/2110.14739/) (2021)
- [CamP: Camera Preconditioning for Neural Radiance Fields](https://mlsanity.com/paper/2308.10902/) (2023)
- [CAP-VSTNet: Content Affinity Preserved Versatile Style Transfer](https://mlsanity.com/paper/2303.17867/) (2023)
- [A Multi-view Multi-task Learning Framework for Multi-variate Time Series Forecasting](https://mlsanity.com/paper/2109.01657/) (2021)
- [An Investigation into Whitening Loss for Self-supervised Learning](https://mlsanity.com/paper/2210.03586/) (2022)
- [PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation](https://mlsanity.com/paper/2410.01680/) (2024)
- [Robust Differentiable SVD](https://mlsanity.com/paper/2104.03821/) (2021)
- [Discovering Universal Geometry in Embeddings with ICA](https://mlsanity.com/paper/2305.13175/) (2023)
- [Sequential Memory with Temporal Predictive Coding](https://mlsanity.com/paper/2305.11982/) (2023)
- [Variance-Covariance Regularization Improves Representation Learning](https://mlsanity.com/paper/2306.13292/) (2023)
- [Stochastic Whitening Batch Normalization](https://mlsanity.com/paper/2106.04413/) (2021)
- [A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models](https://mlsanity.com/paper/2306.00331/) (2023)
- [On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning](https://mlsanity.com/paper/2306.05637/) (2023)
- [DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation](https://mlsanity.com/paper/2403.13322/) (2024)
- [Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory](https://mlsanity.com/paper/2602.09314/) (2026)
- [IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces](https://mlsanity.com/paper/2210.05098/) (2022)
- [Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval](https://mlsanity.com/paper/2312.09716/) (2023)
- [Efficient Deep Learning with Decorrelated Backpropagation](https://mlsanity.com/paper/2405.02385/) (2024)
- [PPISP: Physically-Plausible Compensation and Control of Photometric Variations in Radiance Field Reconstruction](https://mlsanity.com/paper/2601.18336/) (2026)
- [Gauge-invariant representation holonomy](https://mlsanity.com/paper/2601.21653/) (2026)
- [Interpretability in deep learning for finance: a case study for the Heston model](https://mlsanity.com/paper/2104.09476/) (2021)
- [Inverting Adversarially Robust Networks for Image Synthesis](https://mlsanity.com/paper/2106.06927/) (2021)
- [D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory](https://mlsanity.com/paper/2303.00399/) (2023)
- [Interneurons accelerate learning dynamics in recurrent neural networks for statistical adaptation](https://mlsanity.com/paper/2209.10634/) (2022)
- [Attention to the Burstiness in Visual Prompt Tuning!](https://mlsanity.com/paper/2506.22908/) (2025)
- [Posterior Collapse as a Phase Transition in Variational Autoencoders](https://mlsanity.com/paper/2510.01621/) (2025)
- [Beyond Muon: MUD (MomentUm Decorrelation) for Faster Transformer Training](https://mlsanity.com/paper/2603.17970/) (2026)
- [Improving Generalization of Batch Whitening by Convolutional Unit Optimization](https://mlsanity.com/paper/2108.10629/) (2021)

## Abstract (arXiv)

Whitening, or sphering, is a common preprocessing step in statistical analysis to transform random variables to orthogonality. However, due to rotational freedom there are infinitely many possible whitening procedures. Consequently, there is a diverse range of sphering methods in use, for example based on principal component analysis (PCA), Cholesky matrix decomposition and zero-phase component analysis (ZCA), among others. Here we provide an overview of the underlying theory and discuss five natural whitening procedures. Subsequently, we demonstrate that investigating the cross-covariance and the cross-correlation matrix between sphered and original variables allows to break the rotational invariance and to identify optimal whitening transformations. As a result we recommend two particular approaches: ZCA-cor whitening to produce sphered variables that are maximally similar to the original variables, and PCA-cor whitening to obtain sphered variables that maximally compress the original variables.