mlsanityData to 14 Oct 2026

Weight-Entanglement Meets Gradient-Based Neural Architecture Search

arXiv 2312.10440 ↗First posted 16 Dec 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than traditional black-box approaches. In parallel, weight-entanglement has emerged as a technique for more intricate parameter sharing amongst macro-architectural spaces. Since weight-entanglement is not directly compatible with gradient-based NAS methods, these two paradigms have largely developed independently in parallel sub-communities. This paper aims to bridge the gap between these sub-communities by proposing a novel scheme to adapt gradient-based methods for weight-entangled spaces. This enables us to conduct an in-depth comparative assessment and analysis of the performance of gradient-based NAS in weight-entangled search spaces. Our findings reveal that this integration of weight-entanglement and gradient-based NAS brings forth the various benefits of gradient-based methods, while preserving the memory efficiency of weight-entangled spaces. The code for our work is openly accessible https://github.com/automl/TangleNAS.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 16 Dec 2023.

Builds on

33 of 51 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
21 more
  1. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 414
  2. 2018Slimmable Neural Networkscited by 305
  3. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  4. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 260
  5. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  6. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  7. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  8. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 178
  9. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  10. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  11. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  12. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  13. 2020DrNAS: Dirichlet Neural Architecture Searchcited by 73
  14. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  15. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  16. 2019AtomNAS: Fine-Grained End-to-End Neural Architecture Searchcited by 40
  17. 2020Geometry-Aware Gradient Algorithms for Neural Architecture Searchcited by 40
  18. 2019XNAS: Neural Architecture Search with Expert Advicecited by 35
  19. 2022Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognitioncited by 26
  20. 2021Analyzing and Mitigating Interference in Neural Architecture Searchcited by 20
  21. 2022Efficient Architecture Search for Diverse Taskscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024Multi-objective Differentiable Neural Architecture Searchcited by 1
  2. 2025Kernel-Level Energy-Efficient Neural Architecture Search for Tabular Datasetcited by 1
  3. 2025confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methodscited by 0
  4. 2026LLM Compression with Jointly Optimizing Architectural and Quantization choicescited by 0