mlsanityData to 7 Oct 2026

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

arXiv 2109.07383 ↗First posted 15 Sep 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

30 of 47 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2018Self-Attention with Relative Position Representationscited by 1,112
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  10. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  11. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  12. 2017Progressive Neural Architecture Searchcited by 483
18 more
  1. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  2. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 364
  3. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  4. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  5. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  6. 2019The Evolved Transformercited by 180
  7. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  8. 2019Neural Predictor for Neural Architecture Searchcited by 116
  9. 2020BRP-NAS: Prediction-based NAS using GCNscited by 109
  10. 2020A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAScited by 53
  11. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  12. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  13. 2021AutoDropout: Learning Dropout Patterns to Regularize Deep Networkscited by 35
  14. 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
  15. 2019On Neural Architecture Search for Resource-Constrained Hardware Platformscited by 18
  16. 2021Memory-Efficient Differentiable Transformer Architecture Searchcited by 13
  17. 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
  18. 2020Learning Architectures from an Extended Search Space for Language Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 4; 2024: 0; 2025: 2; 2026: 1.

12 papers.

  1. 2022Neural Architecture Search on Efficient Transformers and Beyondcited by 17
  2. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  3. 2022Wide Attention Is The Way Forward For Transformers?cited by 8
  4. 2023RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxiescited by 7
  5. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  6. 2025HEAL: A Hypothesis-Based Preference-Aware Analysis Frameworkcited by 4
  7. 2022DARTFormer: Finding The Best Type Of Attentioncited by 3
  8. 2021The NiuTrans System for the WMT21 Efficiency Taskcited by 2
  9. 2023Improving Autoregressive NLP Tasks via Modular Linearized Attentioncited by 2
  10. 2022Efficient Automatic Machine Learning via Design Graphscited by 0
  11. 2025A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancementcited by 0
  12. 2026Prediction-powered Neural Architecture Searchcited by 0