# MathNAS: If Blocks Have a Role in Mathematical Architecture Design

arXiv: https://arxiv.org/abs/2311.04943 · HTML: https://mlsanity.com/paper/2311.04943/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 51 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search](https://mlsanity.com/paper/1907.01845/) (2019)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [Resnet in Resnet: Generalizing Residual Architectures](https://mlsanity.com/paper/1603.08029/) (2016)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [Blockwisely Supervised Neural Architecture Search with Knowledge Distillation](https://mlsanity.com/paper/1911.13053/) (2019)
- [CARS: Continuous Evolution for Efficient Neural Architecture Search](https://mlsanity.com/paper/1909.04977/) (2019)
- [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020)
- [NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing](https://mlsanity.com/paper/2006.07116/) (2020)
- [Rethinking Channel Dimensions for Efficient Model Design](https://mlsanity.com/paper/2007.00992/) (2020)

## Built on by (most cited first, 4 of 4)

- [Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation](https://mlsanity.com/paper/2409.09324/) (2024)
- [Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signals](https://mlsanity.com/paper/2506.02281/) (2025)
- [A GEN AI Framework for Medical Note Generation](https://mlsanity.com/paper/2410.01841/) (2024)
- [Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports](https://mlsanity.com/paper/2508.15845/) (2025)

## Abstract (arXiv)

Neural Architecture Search (NAS) has emerged as a favoured method for unearthing effective neural architectures. Recent development of large models has intensified the demand for faster search speeds and more accurate search results. However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost. Consider a typical modular search space widely used in NAS, in which a neural architecture consists of m block nodes and a block node has n alternative blocks. Facing the space containing nm candidate networks, existing NAS methods attempt to find the best one by searching and evaluating candidate networks directly.Different from the general strategy that takes architecture search as a whole problem, we propose a novel divide-and-conquer strategy by making use of the modular nature of the search space.Here, we introduce MathNAS, a general NAS framework based on mathematical programming.In MathNAS, the performances of the m*n possible building blocks in the search space are calculated first, and then the performance of a network is directly predicted based on the performances of its building blocks. Although estimating block performances involves network training, just as what happens for network performance evaluation in existing NAS methods, predicting network performance is completely training-free and thus extremely fast. In contrast to the nm candidate networks to evaluate in existing NAS methods, which require training and a formidable computational burden, there are only m*n possible blocks to handle in MathNAS. Therefore, our approach effectively reduces the complexity of network performance evaluation.Our code is available at https://github.com/wangqinsi1/MathNAS.