# BREATHE: Second-Order Gradients and Heteroscedastic Emulation based Design Space Exploration

arXiv: https://arxiv.org/abs/2308.08666 · HTML: https://mlsanity.com/paper/2308.08666/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 56 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 6 of 6)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces](https://mlsanity.com/paper/2109.10964/) (2021)
- [Deep Active Learning with a Neural Architecture Search](https://mlsanity.com/paper/1811.07579/) (2018)
- [CTRL: Clustering Training Losses for Label Error Detection](https://mlsanity.com/paper/2208.08464/) (2022)
- [Towards Robust Detection of Adversarial Examples](https://mlsanity.com/paper/1706.00633/) (2017)
- [EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms](https://mlsanity.com/paper/2303.13745/) (2023)

## Abstract (arXiv)

Researchers constantly strive to explore larger and more complex search spaces in various scientific studies and physical experiments. However, such investigations often involve sophisticated simulators or time-consuming experiments that make exploring and observing new design samples challenging. Previous works that target such applications are typically sample-inefficient and restricted to vector search spaces. To address these limitations, this work proposes a constrained multi-objective optimization (MOO) framework, called BREATHE, that searches not only traditional vector-based design spaces but also graph-based design spaces to obtain best-performing graphs. It leverages second-order gradients and actively trains a heteroscedastic surrogate model for sample-efficient optimization. In a single-objective vector optimization application, it leads to 64.1% higher performance than the next-best baseline, random forest regression. In graph-based search, BREATHE outperforms the next-best baseline, i.e., a graphical version of Gaussian-process-based Bayesian optimization, with up to 64.9% higher performance. In a MOO task, it achieves up to 21.9× higher hypervolume than the state-of-the-art method, multi-objective Bayesian optimization (MOBOpt). BREATHE also outperforms the baseline methods on most standard MOO benchmark applications.