mlsanityData to 7 Oct 2026

Once-for-All: Train One Network and Specialize it for Efficient Deployment

arXiv 1908.09791 ↗First posted 26 Aug 2019cs.LG cs.CV stat.MLcited by 778

Not on the map. Why?

Abstract (arXiv)

We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing CO2 emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks (> 1019) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and CO2 emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting (<600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.

Key facts. Cited by 778 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 184; 2022: 146; 2023: 145; 2024: 112; 2025: 104; 2026: 87.

778 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2021ImageNet-21K Pretraining for the Massescited by 607
  4. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  5. 2022Matryoshka Representation Learningcited by 268
  6. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  8. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  9. 2021Parametric Contrastive Learningcited by 217
  10. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  11. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  12. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
38 more
  1. 2022FlexiViT: One Model for All Patch Sizescited by 151
  2. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  3. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 131
  4. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  5. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  6. 2021Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approachcited by 105
  7. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  8. 2024MobileNetV4 -- Universal Models for the Mobile Ecosystemcited by 98
  9. 2021A Survey of Modern Deep Learning based Object Detection Modelscited by 97
  10. 2021QuantumNAS: Noise-Adaptive Search for Robust Quantum Circuitscited by 97
  11. 2021ResLT: Residual Learning for Long-tailed Recognitioncited by 96
  12. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  13. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 87
  14. 2021Dynamic Slimmable Networkcited by 83
  15. 2021LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Searchcited by 82
  16. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  17. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  18. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  19. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  20. 2025RF-DETR: Neural Architecture Search for Real-Time Detection Transformerscited by 70
  21. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  22. 2021A Comprehensive Survey on Hardware-Aware Neural Architecture Searchcited by 60
  23. 2023SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformercited by 60
  24. 2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
  25. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  26. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  27. 2021Anycost GANs for Interactive Image Synthesis and Editingcited by 54
  28. 2021A Survey on Green Deep Learningcited by 52
  29. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  30. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  31. 2022Generalized Parametric Contrastive Learningcited by 45
  32. 2022GhostNetV2: Enhance Cheap Operation with Long-Range Attentioncited by 44
  33. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
  34. 2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
  35. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  36. 2022Exploring Patch-wise Semantic Relation for Contrastive Learning in Image-to-Image Translation Taskscited by 42
  37. 2022EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Predictioncited by 42
  38. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42

50 most cited of 778 All 778 (JSON)