# Weight-Entanglement Meets Gradient-Based Neural Architecture Search

arXiv: https://arxiv.org/abs/2312.10440 · HTML: https://mlsanity.com/paper/2312.10440/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [DrNAS: Dirichlet Neural Architecture Search](https://mlsanity.com/paper/2006.10355/) (2020)
- [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [AtomNAS: Fine-Grained End-to-End Neural Architecture Search](https://mlsanity.com/paper/1912.09640/) (2019)
- [Geometry-Aware Gradient Algorithms for Neural Architecture Search](https://mlsanity.com/paper/2004.07802/) (2020)
- [XNAS: Neural Architecture Search with Expert Advice](https://mlsanity.com/paper/1906.08031/) (2019)

## Built on by (most cited first, 4 of 4)

- [Multi-objective Differentiable Neural Architecture Search](https://mlsanity.com/paper/2402.18213/) (2024)
- [Kernel-Level Energy-Efficient Neural Architecture Search for Tabular Dataset](https://mlsanity.com/paper/2504.08359/) (2025)
- [confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods](https://mlsanity.com/paper/2507.16533/) (2025)
- [LLM Compression with Jointly Optimizing Architectural and Quantization choices](https://mlsanity.com/paper/2606.04063/) (2026)

## Abstract (arXiv)

Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than traditional black-box approaches. In parallel, weight-entanglement has emerged as a technique for more intricate parameter sharing amongst macro-architectural spaces. Since weight-entanglement is not directly compatible with gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities. This paper aims to bridge the gap between these sub-communities by proposing a novel scheme to adapt gradient-based methods for weight-entangled spaces. This enables us to conduct an in-depth comparative assessment and analysis of the performance of gradient-based NAS in weight-entangled search spaces. Our findings reveal that this integration of weight-entanglement and gradient-based NAS brings forth the various benefits of gradient-based methods, while preserving the memory efficiency of weight-entangled spaces. The code for our work is openly accessible https://github.com/automl/TangleNAS.