# Transfer-Once-For-All: AI Model Optimization for Edge

arXiv: https://arxiv.org/abs/2303.15485 · HTML: https://mlsanity.com/paper/2303.15485/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2007.11330/) (2020)
- [Can weight sharing outperform random architecture search? An investigation with TuNAS](https://mlsanity.com/paper/2008.06120/) (2020)
- [AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling](https://mlsanity.com/paper/2011.09011/) (2020)
- [Neural Architecture Transfer](https://mlsanity.com/paper/2005.05859/) (2020)
- [CompOFA: Compound Once-For-All Networks for Faster Multi-Platform Deployment](https://mlsanity.com/paper/2104.12642/) (2021)
- [K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets](https://mlsanity.com/paper/2106.06442/) (2021)
- [Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms](https://mlsanity.com/paper/2105.03596/) (2021)
- [How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS](https://mlsanity.com/paper/2003.04276/) (2020)
- [Prior-Guided One-shot Neural Architecture Search](https://mlsanity.com/paper/2206.13329/) (2022)

## Built on by (most cited first, 4 of 4)

- [Efficiently Distilling LLMs for Edge Applications](https://mlsanity.com/paper/2404.01353/) (2024)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [Memory-Optimized Once-For-All Network](https://mlsanity.com/paper/2409.05900/) (2024)
- [SEval-NAS: A Search-Agnostic Evaluation for Neural Architecture Search](https://mlsanity.com/paper/2603.00099/) (2026)

## Abstract (arXiv)

Weight-sharing neural architecture search aims to optimize a configurable neural network model (supernet) for a variety of deployment scenarios across many devices with different resource constraints. Existing approaches use evolutionary search to extract models of different sizes from a supernet trained on a very large data set, and then fine-tune the extracted models on the typically small, real-world data set of interest. The computational cost of training thus grows linearly with the number of different model deployment scenarios. Hence, we propose Transfer-Once-For-All (TOFA) for supernet-style training on small data sets with constant computational training cost over any number of edge deployment scenarios. Given a task, TOFA obtains custom neural networks, both the topology and the weights, optimized for any number of edge deployment scenarios. To overcome the challenges arising from small data, TOFA utilizes a unified semi-supervised training loss to simultaneously train all subnets within the supernet, coupled with on-the-fly architecture selection at deployment time.