# Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization

arXiv: https://arxiv.org/abs/2106.11890 · HTML: https://mlsanity.com/paper/2106.11890/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Neural Architecture Search with Bayesian Optimisation and Optimal Transport](https://mlsanity.com/paper/1802.07191/) (2018)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling](https://mlsanity.com/paper/2011.09011/) (2020)
- [Multi-objective Bayesian Optimization using Pareto-frontier Entropy](https://mlsanity.com/paper/1906.00127/) (2019)
- [AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search](https://mlsanity.com/paper/1903.11059/) (2019)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/1609.01051/) (2016)
- [Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2104.07275/) (2021)
- [Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog](https://mlsanity.com/paper/2104.04923/) (2021)
- [Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS](https://mlsanity.com/paper/1911.09336/) (2019)

## Built on by (most cited first, 15 of 15)

- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces](https://mlsanity.com/paper/2109.10964/) (2021)
- [Pareto Set Learning for Expensive Multi-Objective Optimization](https://mlsanity.com/paper/2210.08495/) (2022)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [Robust Multi-Objective Bayesian Optimization Under Input Noise](https://mlsanity.com/paper/2202.07549/) (2022)
- [Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge](https://mlsanity.com/paper/2509.09955/) (2025)
- [Supernet Training for Federated Image Classification under System Heterogeneity](https://mlsanity.com/paper/2206.01366/) (2022)
- [Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search](https://mlsanity.com/paper/2206.13130/) (2022)
- [Adaptive Pareto-Optimal Token Merging for Edge Transformer Models in Semantic Communication](https://mlsanity.com/paper/2509.09168/) (2025)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Subgraph Stationary Hardware-Software Inference Co-Design](https://mlsanity.com/paper/2306.17266/) (2023)
- [MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment](https://mlsanity.com/paper/2603.15954/) (2026)
- [Dynamical Isometry based Rigorous Fair Neural Architecture Search](https://mlsanity.com/paper/2307.02263/) (2023)
- [Dense Optimizer : An Information Entropy-Guided Structural Search Method for Dense-like Neural Network Design](https://mlsanity.com/paper/2410.07499/) (2024)
- [Optimising for Energy Efficiency and Performance in Machine Learning](https://mlsanity.com/paper/2601.08991/) (2026)

## Abstract (arXiv)

When tuning the architecture and hyperparameters of large machine learning models for on-device deployment, it is desirable to understand the optimal trade-offs between on-device latency and model accuracy. In this work, we leverage recent methodological advances in Bayesian optimization over high-dimensional search spaces and multi-objective Bayesian optimization to efficiently explore these trade-offs for a production-scale on-device natural language understanding model at Facebook.