# ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradient Accumulation

arXiv: https://arxiv.org/abs/2011.11233 · HTML: https://mlsanity.com/paper/2011.11233/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 3 of 3)

- [Generative AI for the Optimization of Next-Generation Wireless Networks: Basics, State-of-the-Art, and Open Challenges](https://mlsanity.com/paper/2405.17454/) (2024)
- [XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference](https://mlsanity.com/paper/2404.15420/) (2024)
- [Robust MAE-Driven NAS: From Mask Reconstruction to Architecture Innovation](https://mlsanity.com/paper/2311.12086/) (2023)

## Abstract (arXiv)

Albeit being a prevalent architecture searching approach, differentiable architecture search (DARTS) is largely hindered by its substantial memory cost since the entire supernet resides in the memory. This is where the single-path DARTS comes in, which only chooses a single-path submodel at each step. While being memory-friendly, it also comes with low computational costs. Nonetheless, we discover a critical issue of single-path DARTS that has not been primarily noticed. Namely, it also suffers from severe performance collapse since too many parameter-free operations like skip connections are derived, just like DARTS does. In this paper, we propose a new algorithm called RObustifying Memory-Efficient NAS (ROME) to give a cure. First, we disentangle the topology search from the operation search to make searching and evaluation consistent. We then adopt Gumbel-Top2 reparameterization and gradient accumulation to robustify the unwieldy bi-level optimization. We verify ROME extensively across 15 benchmarks to demonstrate its effectiveness and robustness.