# CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment

arXiv: https://arxiv.org/abs/2104.12642 · HTML: https://mlsanity.com/paper/2104.12642/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 23 of 23)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [On Network Design Spaces for Visual Recognition](https://mlsanity.com/paper/1905.13214/) (2019)
- [AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling](https://mlsanity.com/paper/2011.09011/) (2020)
- [AOWS: Adaptive and optimal network width search with latency constraints](https://mlsanity.com/paper/2005.10481/) (2020)

## Built on by (most cited first, 26 of 26)

- [SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads](https://mlsanity.com/paper/2312.16733/) (2023)
- [Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing](https://mlsanity.com/paper/2210.04153/) (2022)
- [One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2111.01203/) (2021)
- [DetOFA: Efficient Training of Once-for-All Networks for Object Detection Using Path Filter](https://mlsanity.com/paper/2303.13121/) (2023)
- [SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference](https://mlsanity.com/paper/2301.10879/) (2023)
- [Transfer-Once-For-All: AI Model Optimization for Edge](https://mlsanity.com/paper/2303.15485/) (2023)
- [Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers](https://mlsanity.com/paper/2407.18175/) (2024)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [Searching for TrioNet: Combining Convolution with Local and Global Self-Attention](https://mlsanity.com/paper/2111.07547/) (2021)
- [Pareto-Secure Machine Learning (PSML): Fingerprinting and Securing Inference Serving Systems](https://mlsanity.com/paper/2307.01292/) (2023)
- [Prioritized Subnet Sampling for Resource-Adaptive Supernet Training](https://mlsanity.com/paper/2109.05432/) (2021)
- [A Hardware-Aware System for Accelerating Deep Neural Network Optimization](https://mlsanity.com/paper/2202.12954/) (2022)
- [Switchable Representation Learning Framework with Self-compatibility](https://mlsanity.com/paper/2206.08289/) (2022)
- [Subgraph Stationary Hardware-Software Inference Co-Design](https://mlsanity.com/paper/2306.17266/) (2023)
- [DεpS: Delayed ε-Shrinking for Faster Once-For-All Training](https://mlsanity.com/paper/2407.06167/) (2024)
- [Memory-Optimized Once-For-All Network](https://mlsanity.com/paper/2409.05900/) (2024)
- [KLAS: Using Similarity to Stitch Neural Networks for Improved Accuracy-Efficiency Tradeoffs](https://mlsanity.com/paper/2605.29259/) (2026)
- [ParaDiS: Parallelly Distributable Slimmable Neural Networks](https://mlsanity.com/paper/2110.02724/) (2021)
- [Does Interference Exist When Training a Once-For-All Network?](https://mlsanity.com/paper/2204.09210/) (2022)
- [Bespoke: A Block-Level Neural Network Optimization Framework for Low-Cost Deployment](https://mlsanity.com/paper/2303.01913/) (2023)
- [MatchNAS: Optimizing Edge AI in Sparse-Label Data Contexts via Automating Deep Neural Network Porting for Mobile Deployment](https://mlsanity.com/paper/2402.13525/) (2024)
- [SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model](https://mlsanity.com/paper/2406.00195/) (2024)
- [FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learning](https://mlsanity.com/paper/2504.12849/) (2025)
- [ProARD: progressive adversarial robustness distillation: provide wide range of robust students](https://mlsanity.com/paper/2506.07666/) (2025)
- [AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devices](https://mlsanity.com/paper/2511.22355/) (2025)
- [A Generalized Optimization Engine (GOE) for Edge AI Inference Acceleration](https://mlsanity.com/paper/2608.28652/) (2026)

## Abstract (arXiv)

The emergence of CNNs in mainstream deployment has necessitated methods to design and train efficient architectures tailored to maximize the accuracy under diverse hardware & latency constraints. To scale these resource-intensive tasks with an increasing number of deployment targets, Once-For-All (OFA) proposed an approach to jointly train several models at once with a constant training cost. However, this cost remains as high as 40-50 GPU days and also suffers from a combinatorial explosion of sub-optimal model configurations. We seek to reduce this search space -- and hence the training budget -- by constraining search to models close to the accuracy-latency Pareto frontier. We incorporate insights of compound relationships between model dimensions to build CompOFA, a design space smaller by several orders of magnitude. Through experiments on ImageNet, we demonstrate that even with simple heuristics we can achieve a 2x reduction in training time and 216x speedup in model search/extraction time compared to the state of the art, without loss of Pareto optimality! We also show that this smaller design space is dense enough to support equally accurate models for a similar diversity of hardware and latency targets, while also reducing the complexity of the training and subsequent extraction algorithms.