# Robustness via Deep Low-Rank Representations

arXiv: https://arxiv.org/abs/1804.07090 · HTML: https://mlsanity.com/paper/1804.07090/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2018.

## Built on by (most cited first, 11 of 11)

- [Make Some Noise: Reliable and Efficient Single-Step Adversarial Training](https://mlsanity.com/paper/2202.01181/) (2022)
- [An Empirical Study of Implicit Regularization in Deep Offline RL](https://mlsanity.com/paper/2207.02099/) (2022)
- [CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection](https://mlsanity.com/paper/2102.05311/) (2021)
- [Understanding Robust Learning through the Lens of Representation Similarities](https://mlsanity.com/paper/2206.09868/) (2022)
- [Projected Randomized Smoothing for Certified Adversarial Robustness](https://mlsanity.com/paper/2309.13794/) (2023)
- [A law of adversarial risk, interpolation, and label noise](https://mlsanity.com/paper/2207.03933/) (2022)
- [Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences](https://mlsanity.com/paper/2112.05359/) (2021)
- [Calibrate and Debias Layer-wise Sampling for Graph Convolutional Networks](https://mlsanity.com/paper/2206.00583/) (2022)
- [One Rank at a Time: Cascading Error Dynamics in Sequential Learning](https://mlsanity.com/paper/2505.22602/) (2025)
- [Robust and Information-theoretically Safe Bias Classifier against Adversarial Attacks](https://mlsanity.com/paper/2111.04404/) (2021)
- [Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks](https://mlsanity.com/paper/2508.00043/) (2025)

## Abstract (arXiv)

We investigate the effect of the dimensionality of the representations learned in Deep Neural Networks (DNNs) on their robustness to input perturbations, both adversarial and random. To achieve low dimensionality of learned representations, we propose an easy-to-use, end-to-end trainable, low-rank regularizer (LR) that can be applied to any intermediate layer representation of a DNN. This regularizer forces the feature representations to (mostly) lie in a low-dimensional linear subspace. We perform a wide range of experiments that demonstrate that the LR indeed induces low rank on the representations, while providing modest improvements to accuracy as an added benefit. Furthermore, the learned features make the trained model significantly more robust to input perturbations such as Gaussian and adversarial noise (even without adversarial training). Lastly, the low-dimensionality means that the learned features are highly compressible; thus discriminative features of the data can be stored using very little memory. Our experiments indicate that models trained using the LR learn robust classifiers by discovering subspaces that avoid non-robust features. Algorithmically, the LR is scalable, generic, and straightforward to implement into existing deep learning frameworks.