# Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits

arXiv: https://arxiv.org/abs/2302.01888 · HTML: https://mlsanity.com/paper/2302.01888/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 16 of 16)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Genetic CNN](https://mlsanity.com/paper/1703.01513/) (2017)
- [Overcoming Multi-Model Forgetting](https://mlsanity.com/paper/1902.08232/) (2019)
- [POPNASv3: a Pareto-Optimal Neural Architecture Search Solution for Image and Time Series Classification](https://mlsanity.com/paper/2212.06735/) (2022)
- [POPNASv2: An Efficient Multi-Objective Neural Architecture Search Technique](https://mlsanity.com/paper/2210.02959/) (2022)
- [Anticipate, Ensemble and Prune: Improving Convolutional Neural Networks via Aggregated Early Exits](https://mlsanity.com/paper/2301.12168/) (2023)

## Built on by (most cited first, 3 of 3)

- [NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networks](https://mlsanity.com/paper/2401.13330/) (2024)
- [POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator](https://mlsanity.com/paper/2409.20447/) (2024)
- [Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.00960/) (2023)

## Abstract (arXiv)

The use of Neural Architecture Search (NAS) techniques to automate the design of neural networks has become increasingly popular in recent years. The proliferation of devices with different hardware characteristics using such neural networks, as well as the need to reduce the power consumption for their search, has led to the realisation of Once-For-All (OFA), an eco-friendly algorithm characterised by the ability to generate easily adaptable models through a single learning process. In order to improve this paradigm and develop high-performance yet eco-friendly NAS techniques, this paper presents OFAv2, the extension of OFA aimed at improving its performance while maintaining the same ecological advantage. The algorithm is improved from an architectural point of view by including early exits, parallel blocks and dense skip connections. The training process is extended by two new phases called Elastic Level and Elastic Height. A new Knowledge Distillation technique is presented to handle multi-output networks, and finally a new strategy for dynamic teacher network selection is proposed. These modifications allow OFAv2 to improve its accuracy performance on the Tiny ImageNet dataset by up to 12.07% compared to the original version of OFA, while maintaining the algorithm flexibility and advantages.