# Contrastive Neural Architecture Search with Neural Architecture Comparators

arXiv: https://arxiv.org/abs/2103.05471 · HTML: https://mlsanity.com/paper/2103.05471/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 59 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)

## Built on by (most cited first, 30 of 34)

- [ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency](https://mlsanity.com/paper/2106.02342/) (2021)
- [Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting](https://mlsanity.com/paper/2211.16126/) (2022)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Improving Robustness by Enhancing Weak Subnets](https://mlsanity.com/paper/2201.12765/) (2022)
- [NAR-Former: Neural Architecture Representation Learning towards Holistic Attributes Prediction](https://mlsanity.com/paper/2211.08024/) (2022)
- [Fully Automated Correlated Time Series Forecasting in Minutes](https://mlsanity.com/paper/2411.05833/) (2024)
- [SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS](https://mlsanity.com/paper/2403.04161/) (2024)
- [FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer](https://mlsanity.com/paper/2403.12821/) (2024)
- [Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images](https://mlsanity.com/paper/2211.10643/) (2022)
- [Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2109.07582/) (2021)
- [NAR-Former V2: Rethinking Transformer for Universal Neural Network Representation Learning](https://mlsanity.com/paper/2306.10792/) (2023)
- [Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search](https://mlsanity.com/paper/2201.12725/) (2022)
- [Boosting Semi-Supervised Learning with Contrastive Complementary Labeling](https://mlsanity.com/paper/2212.06643/) (2022)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [Building Optimal Neural Architectures using Interpretable Knowledge](https://mlsanity.com/paper/2403.13293/) (2024)
- [CAP: A Context-Aware Neural Predictor for NAS](https://mlsanity.com/paper/2406.02056/) (2024)
- [Bi-fidelity Evolutionary Multiobjective Search for Adversarially Robust Deep Neural Architectures](https://mlsanity.com/paper/2207.05321/) (2022)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)
- [DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning](https://mlsanity.com/paper/2302.13020/) (2023)
- [NN-Former: Rethinking Graph Structure in Neural Architecture Representation](https://mlsanity.com/paper/2507.00880/) (2025)
- [Differentiable Search of Accurate and Robust Architectures](https://mlsanity.com/paper/2212.14049/) (2022)
- [How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor](https://mlsanity.com/paper/2311.18451/) (2023)
- [NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks](https://mlsanity.com/paper/2302.00059/) (2023)
- [Neural Architecture Retrieval](https://mlsanity.com/paper/2307.07919/) (2023)
- [Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding](https://mlsanity.com/paper/2510.18360/) (2025)
- [WPNAS: Neural Architecture Search by jointly using Weight Sharing and Predictor](https://mlsanity.com/paper/2203.02086/) (2022)
- [MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency](https://mlsanity.com/paper/2310.15074/) (2023)
- [Guided Evolution with Binary Discriminators for ML Program Search](https://mlsanity.com/paper/2402.05821/) (2024)
- [FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search](https://mlsanity.com/paper/2404.15622/) (2024)
- [Language Embedding Meets Dynamic Graph: A New Exploration for Neural Architecture Representation Learning](https://mlsanity.com/paper/2506.07735/) (2025)

## Abstract (arXiv)

One of the key steps in Neural Architecture Search (NAS) is to estimate the performance of candidate architectures. Existing methods either directly use the validation performance or learn a predictor to estimate the performance. However, these methods can be either computationally expensive or very inaccurate, which may severely affect the search efficiency and performance. Moreover, as it is very difficult to annotate architectures with accurate performance on specific tasks, learning a promising performance predictor is often non-trivial due to the lack of labeled data. In this paper, we argue that it may not be necessary to estimate the absolute performance for NAS. On the contrary, we may need only to understand whether an architecture is better than a baseline one. However, how to exploit this comparison information as the reward and how to well use the limited labeled data remains two great challenges. In this paper, we propose a novel Contrastive Neural Architecture Search (CTNAS) method which performs architecture search by taking the comparison results between architectures as the reward. Specifically, we design and learn a Neural Architecture Comparator (NAC) to compute the probability of candidate architectures being better than a baseline one. Moreover, we present a baseline updating scheme to improve the baseline iteratively in a curriculum learning manner. More critically, we theoretically show that learning NAC is equivalent to optimizing the ranking over architectures. Extensive experiments in three search spaces demonstrate the superiority of our CTNAS over existing methods.