# MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation

arXiv: https://arxiv.org/abs/2003.12238 · HTML: https://mlsanity.com/paper/2003.12238/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2020.

## Built on by (most cited first, 30 of 57)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks](https://mlsanity.com/paper/2104.08815/) (2021)
- [From Distributed Machine Learning to Federated Learning: A Survey](https://mlsanity.com/paper/2104.14362/) (2021)
- [Lower Bounds and Accelerated Algorithms for Bilevel Optimization](https://mlsanity.com/paper/2102.03926/) (2021)
- [PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation](https://mlsanity.com/paper/2108.10710/) (2021)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [KNAS: Green Neural Architecture Search](https://mlsanity.com/paper/2111.13293/) (2021)
- [FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks](https://mlsanity.com/paper/2111.11066/) (2021)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)
- [Federated Learning for Internet of Things: A Federated Learning Framework for On-device Anomaly Data Detection](https://mlsanity.com/paper/2106.07976/) (2021)
- [Linear, or Non-Linear, That is the Question!](https://mlsanity.com/paper/2111.07265/) (2021)
- [PreGenie: An Agentic Framework for High-quality Visual Presentation Generation](https://mlsanity.com/paper/2505.21660/) (2025)
- [DropNAS: Grouped Operation Dropout for Differentiable Architecture Search](https://mlsanity.com/paper/2201.11679/) (2022)
- [Self-supervised Cross-silo Federated Neural Architecture Search](https://mlsanity.com/paper/2101.11896/) (2021)
- [FedMe: Federated Learning via Model Exchange](https://mlsanity.com/paper/2110.07868/) (2021)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [Fisher Task Distance and Its Application in Neural Architecture Search](https://mlsanity.com/paper/2103.12827/) (2021)
- [SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models](https://mlsanity.com/paper/2502.20422/) (2025)
- [Learning Calibrated-Guidance for Object Detection in Aerial Images](https://mlsanity.com/paper/2103.11399/) (2021)
- [L2NAS: Learning to Optimize Neural Architectures via Continuous-Action Reinforcement Learning](https://mlsanity.com/paper/2109.12425/) (2021)
- [Automatic Relation-aware Graph Network Proliferation](https://mlsanity.com/paper/2205.15678/) (2022)
- [TransNAS-TSAD: Harnessing Transformers for Multi-Objective Neural Architecture Search in Time Series Anomaly Detection](https://mlsanity.com/paper/2311.18061/) (2023)
- [Differentiable Combinatorial Scheduling at Scale](https://mlsanity.com/paper/2406.06593/) (2024)
- [Neural Architecture Search via Combinatorial Multi-Armed Bandit](https://mlsanity.com/paper/2101.00336/) (2021)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2104.07145/) (2021)
- [Differentiable Neural Architecture Search for Extremely Lightweight Image Super-Resolution](https://mlsanity.com/paper/2105.03939/) (2021)
- [FocusFormer: Focusing on What We Need via Architecture Sampler](https://mlsanity.com/paper/2208.10861/) (2022)
- [Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks](https://mlsanity.com/paper/2209.14252/) (2022)

## Abstract (arXiv)

Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of second-order methods. In this paper, we demonstrate that gradient errors caused by such approximations lead to suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. To remedy this, this paper proposes mldas, a mixed-level reformulation for NAS that can be optimized efficiently and reliably. It is shown that even when using a simple first-order method on the mixed-level formulation, mldas can achieve a lower validation error for NAS problems. Consequently, architectures obtained by our method achieve consistently higher accuracies than those obtained from bilevel optimization. Moreover, mldas proposes a framework beyond DARTS. It is upgraded via model size-based search and early stopping strategies to complete the search process in around 5 hours. Extensive experiments within the convolutional architecture search space validate the effectiveness of our approach.