# Learning Compressed Transforms with Low Displacement Rank

arXiv: https://arxiv.org/abs/1810.02309 · HTML: https://mlsanity.com/paper/1810.02309/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2018.

## Built on by (most cited first, 19 of 19)

- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networks](https://mlsanity.com/paper/2409.11772/) (2024)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Spectraformer: A Unified Random Feature Framework for Transformer](https://mlsanity.com/paper/2405.15310/) (2024)
- [Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2406.17740/) (2024)
- [Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2110.09374/) (2021)
- [Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformers](https://mlsanity.com/paper/2406.15881/) (2024)
- [Block Circulant Adapter for Large Language Models](https://mlsanity.com/paper/2505.00582/) (2025)
- [Intrinisic Gradient Compression for Federated Learning](https://mlsanity.com/paper/2112.02656/) (2021)
- [Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets](https://mlsanity.com/paper/2303.06797/) (2023)
- [Arithmetic Circuits, Structured Matrices and (not so) Deep Learning](https://mlsanity.com/paper/2206.12490/) (2022)
- [ToTMNet: FFT-Accelerated Toeplitz Temporal Mixing Network for Lightweight Remote Photoplethysmography](https://mlsanity.com/paper/2601.04159/) (2026)
- [Neural-HSS: Hierarchical Semi-Separable Neural PDE Solver](https://mlsanity.com/paper/2602.18248/) (2026)
- [HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation](https://mlsanity.com/paper/2110.15440/) (2021)
- [Reducing Memory Requirements for the IPU using Butterfly Factorizations](https://mlsanity.com/paper/2309.08946/) (2023)
- [JustDense: Just using Dense instead of Sequence Mixer for Time Series analysis](https://mlsanity.com/paper/2508.09153/) (2025)
- [Customizing the Inductive Biases of Softmax Attention using Structured Matrices](https://mlsanity.com/paper/2509.07963/) (2025)
- [Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count](https://mlsanity.com/paper/2605.08171/) (2026)

## Abstract (arXiv)

The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR matrices with more general displacement operators, and explicitly learn over both the operators and the low-rank component. This class generalizes several previous constructions while preserving compression and efficient computation. We prove bounds on the VC dimension of multi-layer neural networks with structured weight matrices and show empirically that our compact parameterization can reduce the sample complexity of learning. When replacing weight layers in fully-connected, convolutional, and recurrent neural networks for image classification and language modeling tasks, our new classes exceed the accuracy of existing compression approaches, and on some tasks also outperform general unstructured layers while using more than 20x fewer parameters.