mlsanityData to 7 Oct 2026

Speeding up Convolutional Neural Networks with Low Rank Expansions

arXiv 1405.3866 ↗First posted 15 May 2014cs.CVcited by 283

Not on the map. Why?

Abstract (arXiv)

The focus of this paper is speeding up the evaluation of convolutional neural networks. While delivering impressive results across a range of computer vision and machine learning tasks, these networks are computationally demanding, limiting their deployability. Convolutional layers generally consume the bulk of the processing time, and so in this work we present two simple schemes for drastically speeding up these layers. This is achieved by exploiting cross-channel or filter redundancy to construct a low rank basis of filters that are rank-1 in the spatial domain. Our methods are architecture agnostic, and can be easily applied to existing CPU and GPU convolutional frameworks for tuneable speedup performance. We demonstrate this with a real world network designed for scene text character recognition, showing a possible 2.5x speedup with no loss in accuracy, and 4.5x speedup with less than 1% drop in accuracy, still achieving state-of-the-art on standard benchmarks.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 May 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 80; 2022: 55; 2023: 44; 2024: 42; 2025: 37; 2026: 24.

283 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Dynamic Neural Networks: A Surveycited by 405
  3. 2021NeRV: Neural Representations for Videoscited by 255
  4. 2021Concealed Object Detectioncited by 212
  5. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  6. 2022Language model compression with weighted low-rank factorizationcited by 192
  7. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  8. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  9. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  10. 2021Involution: Inverting the Inherence of Convolution for Visual Recognitioncited by 99
  11. 2021The Efficiency Misnomercited by 98
  12. 2024DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Modelscited by 98
38 more
  1. 2021Trustworthy AI: A Computational Perspectivecited by 93
  2. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  3. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 66
  4. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  5. 2021CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretationcited by 57
  6. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  7. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  8. 2022Revisiting Random Channel Pruning for Neural Network Compressioncited by 50
  9. 2022Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Modelscited by 44
  10. 2022GhostNetV2: Enhance Cheap Operation with Long-Range Attentioncited by 44
  11. 2024A Survey on Transformer Compressioncited by 44
  12. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  13. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 42
  14. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  15. 2022NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Resultscited by 40
  16. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 37
  17. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  18. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  19. 2021Skip-Convolutions for Efficient Video Processingcited by 32
  20. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  21. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  22. 2024How Far Can We Compress Instant-NGP-Based NeRF?cited by 30
  23. 2024CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networkscited by 29
  24. 2021Content-Aware GAN Compressioncited by 28
  25. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  26. 2022projUNN: efficient method for training deep networks with unitary matricescited by 27
  27. 2021Selfish Sparse RNN Trainingcited by 26
  28. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  29. 2021Distribution-sensitive Information Retention for Accurate Binary Neural Networkcited by 25
  30. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  31. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  32. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  33. 2022Data-Efficient Structured Pruning via Submodular Optimizationcited by 22
  34. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  35. 2024Spectral Adapter: Fine-Tuning in Spectral Spacecited by 21
  36. 2022Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptionscited by 20
  37. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  38. 2021Scaling Up Exact Neural Network Compression by ReLU Stabilitycited by 19

50 most cited of 283 All 283 (JSON)