# Structured Transforms for Small-Footprint Deep Learning

arXiv: https://arxiv.org/abs/1510.01722 · HTML: https://mlsanity.com/paper/1510.01722/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2015.

## Built on by (most cited first, 30 of 42)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Simple Hardware-Efficient Long Convolutions for Sequence Modeling](https://mlsanity.com/paper/2302.06646/) (2023)
- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores](https://mlsanity.com/paper/2311.05908/) (2023)
- [Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN](https://mlsanity.com/paper/2107.14444/) (2021)
- [A Survey on Symbolic Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2408.10210/) (2024)
- [ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression](https://mlsanity.com/paper/2106.02241/) (2021)
- [GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09650/) (2021)
- [Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networks](https://mlsanity.com/paper/2409.11772/) (2024)
- [MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning](https://mlsanity.com/paper/2503.18368/) (2025)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11438/) (2021)
- [Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning](https://mlsanity.com/paper/2406.17740/) (2024)
- [SENetV2: Aggregated dense layer for channelwise and global representations](https://mlsanity.com/paper/2311.10807/) (2023)
- [Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformers](https://mlsanity.com/paper/2406.15881/) (2024)
- [TropNNC: Structured Neural Network Compression Using Tropical Geometry](https://mlsanity.com/paper/2409.03945/) (2024)
- [Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers](https://mlsanity.com/paper/2111.14836/) (2021)
- [What is Lost in Knowledge Distillation?](https://mlsanity.com/paper/2311.04142/) (2023)
- [Composing Linear Layers from Irreducibles](https://mlsanity.com/paper/2507.11688/) (2025)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2601.22563/) (2026)
- [Arithmetic Circuits, Structured Matrices and (not so) Deep Learning](https://mlsanity.com/paper/2206.12490/) (2022)
- [Enhancing Split Computing and Early Exit Applications through Predefined Sparsity](https://mlsanity.com/paper/2407.11763/) (2024)
- [ToTMNet: FFT-Accelerated Toeplitz Temporal Mixing Network for Lightweight Remote Photoplethysmography](https://mlsanity.com/paper/2601.04159/) (2026)
- [Squeeze aggregated excitation network](https://mlsanity.com/paper/2308.13343/) (2023)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2410.09771/) (2024)

## Abstract (arXiv)

We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices. We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank. Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured. Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques. In keyword spotting applications in mobile speech recognition, our methods are much more effective than standard linear low-rank bottleneck layers and nearly retain the performance of state of the art models, while providing more than 3.5-fold compression.