# SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation

arXiv: https://arxiv.org/abs/2407.20062 · HTML: https://mlsanity.com/paper/2407.20062/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [Blockwisely Supervised Neural Architecture Search with Knowledge Distillation](https://mlsanity.com/paper/1911.13053/) (2019)
- [AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling](https://mlsanity.com/paper/2011.09011/) (2020)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [Evolving Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1710.10741/) (2017)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [Contextual Encoder-Decoder Network for Visual Saliency Prediction](https://mlsanity.com/paper/1902.06634/) (2019)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition](https://mlsanity.com/paper/2208.05768/) (2022)
- [TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction](https://mlsanity.com/paper/2301.02315/) (2023)
- [A Dilated Inception Network for Visual Saliency Prediction](https://mlsanity.com/paper/1904.03571/) (2019)
- [Tidying Deep Saliency Prediction Architectures](https://mlsanity.com/paper/2003.04942/) (2020)

## Built on by (most cited first, 1 of 1)

- [Modeling Saliency Dataset Bias](https://mlsanity.com/paper/2505.10169/) (2025)

## Abstract (arXiv)

Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction. However, the manual configuration of the neural network architectures requires domain knowledge expertise and can still be time-consuming and error-prone. To solve this, we propose a new Neural Architecture Search (NAS) framework for saliency prediction with two contributions. Firstly, a supernet for saliency prediction is built with a weight-sharing network containing all candidate architectures, by integrating a dynamic convolution into the encoder-decoder in the supernet, termed SalNAS. Secondly, despite the fact that SalNAS is highly efficient (20.98 million parameters), it can suffer from the lack of generalization. To solve this, we propose a self-knowledge distillation approach, termed Self-KD, that trains the student SalNAS with the weighted average information between the ground truth and the prediction from the teacher model. The teacher model, while sharing the same architecture, contains the best-performing weights chosen by cross-validation. Self-KD can generalize well without the need to compute the gradient in the teacher model, enabling an efficient training system. By utilizing Self-KD, SalNAS outperforms other state-of-the-art saliency prediction models in most evaluation rubrics across seven benchmark datasets while being a lightweight model. The code will be available at https://github.com/chakkritte/SalNAS