# RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

arXiv: https://arxiv.org/abs/2109.07383 · HTML: https://mlsanity.com/paper/2109.07383/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS](https://mlsanity.com/paper/2004.01899/) (2020)
- [Improving Transformer Models by Reordering their Sublayers](https://mlsanity.com/paper/1911.03864/) (2019)
- [AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search](https://mlsanity.com/paper/2001.04246/) (2020)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Weight Distillation: Transferring the Knowledge in Neural Network Parameters](https://mlsanity.com/paper/2009.09152/) (2020)
- [On Neural Architecture Search for Resource-Constrained Hardware Platforms](https://mlsanity.com/paper/1911.00105/) (2019)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [An Efficient Transformer Decoder with Compressed Sub-layers](https://mlsanity.com/paper/2101.00542/) (2021)
- [Learning Architectures from an Extended Search Space for Language Modeling](https://mlsanity.com/paper/2005.02593/) (2020)

## Built on by (most cited first, 12 of 12)

- [Neural Architecture Search on Efficient Transformers and Beyond](https://mlsanity.com/paper/2207.13955/) (2022)
- [ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation](https://mlsanity.com/paper/2308.02223/) (2023)
- [Wide Attention Is The Way Forward For Transformers?](https://mlsanity.com/paper/2210.00640/) (2022)
- [RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies](https://mlsanity.com/paper/2301.09850/) (2023)
- [Learning Evaluation Models from Large Language Models for Sequence Generation](https://mlsanity.com/paper/2308.04386/) (2023)
- [HEAL: A Hypothesis-Based Preference-Aware Analysis Framework](https://mlsanity.com/paper/2508.19922/) (2025)
- [DARTFormer: Finding The Best Type Of Attention](https://mlsanity.com/paper/2210.00641/) (2022)
- [The NiuTrans System for the WMT21 Efficiency Task](https://mlsanity.com/paper/2109.08003/) (2021)
- [Improving Autoregressive NLP Tasks via Modular Linearized Attention](https://mlsanity.com/paper/2304.08453/) (2023)
- [Efficient Automatic Machine Learning via Design Graphs](https://mlsanity.com/paper/2210.12257/) (2022)
- [A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancement](https://mlsanity.com/paper/2507.14200/) (2025)
- [Prediction-powered Neural Architecture Search](https://mlsanity.com/paper/2610.01317/) (2026)

## Abstract (arXiv)

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.