mlsanityData to 14 Oct 2026

Structured Transforms for Small-Footprint Deep Learning

arXiv 1510.01722 ↗First posted 6 Oct 2015stat.ML cs.CV cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices. We propose a unified framework to learn a broad family of structured parameter matrices that are characterized by the notion of low displacement rank. Our structured transforms admit fast function and gradient evaluation, and span a rich range of parameter sharing configurations whose statistical modeling capacity can be explicitly tuned along a continuum from structured to unstructured. Experimental results show that these transforms can significantly accelerate inference and forward/backward passes during training, and offer superior accuracy-compactness-speed tradeoffs in comparison to a number of existing techniques. In keyword spotting applications in mobile speech recognition, our methods are much more effective than standard linear low-rank bottleneck layers and nearly retain the performance of state of the art models, while providing more than 3.5-fold compression.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Oct 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 3; 2023: 6; 2024: 7; 2025: 6; 2026: 11.

42 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  3. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  4. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  5. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  6. 2021CHIP: CHannel Independence-based Pruning for Compact Neural Networkscited by 78
  7. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  8. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 46
  9. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 33
  10. 2021Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNNcited by 9
  11. 2024A Survey on Symbolic Knowledge Distillation of Large Language Modelscited by 9
  12. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compressioncited by 8
30 more
  1. 2021GST: Group-Sparse Training for Accelerating Deep Reinforcement Learningcited by 7
  2. 2024Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networkscited by 7
  3. 2025MoST: Efficient Monarch Sparse Tuning for 3D Representation Learningcited by 6
  4. 2021Doping: A technique for efficient compression of LSTM models using sparse structured additive matricescited by 5
  5. 2021Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognitioncited by 5
  6. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  7. 2023SENetV2: Aggregated dense layer for channelwise and global representationscited by 3
  8. 2024Fast Tree-Field Integrators: From Low Displacement Rank to Topological Transformerscited by 3
  9. 2024TropNNC: Structured Neural Network Compression Using Tropical Geometrycited by 3
  10. 2021Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multiplierscited by 2
  11. 2023What is Lost in Knowledge Distillation?cited by 2
  12. 2025Composing Linear Layers from Irreduciblescited by 2
  13. 2026EUGens: Efficient, Unified, and General Dense Layerscited by 2
  14. 2022Arithmetic Circuits, Structured Matrices and (not so) Deep Learningcited by 1
  15. 2024Enhancing Split Computing and Early Exit Applications through Predefined Sparsitycited by 1
  16. 2026ToTMNet: FFT-Accelerated Toeplitz Temporal Mixing Network for Lightweight Remote Photoplethysmographycited by 1
  17. 2023Squeeze aggregated excitation networkcited by 0
  18. 2024EUGens: Efficient, Unified, and General Dense Layerscited by 0
  19. 2025NdLinear: Preserving Multi-Dimensional Structure for Parameter-Efficient Neural Networkscited by 0
  20. 2025Penrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptationcited by 0
  21. 2025Element-wise Modulation of Random Matrices for Efficient Neural Layerscited by 0
  22. 2026EdgeLDR: Quaternion Low-Displacement Rank Neural Networks for Edge-Efficient Deep Learningcited by 0
  23. 2026Once-for-All Channel Mixers (HYPERTINYPW): Generative Compression for TinyMLcited by 0
  24. 2026Leveraging Kernel Symmetry for Joint Compression and Error Mitigation in Edge Model Transfercited by 0
  25. 2026Learning from Compressed CT: Feature Attention Style Transfer and Structured Factorized Projections for Resource-Efficient Medical Image Analysiscited by 0
  26. 2026Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Countcited by 0
  27. 2026Two-Valued Symmetric Circulant Matrices: Applications in Deep Learningcited by 0
  28. 2026Linear Reusable Neural Bases Architecture for Network Compressioncited by 0
  29. 2026Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matricescited by 0
  30. 2026Rationally Linearized MLPscited by 0