# Human-Aided Saliency Maps Improve Generalization of Deep Learning

arXiv: https://arxiv.org/abs/2105.03492 · HTML: https://mlsanity.com/paper/2105.03492/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 7 May 2021.

## Builds on (most cited first, 23 of 23)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Learning from Synthetic Data for Crowd Counting in the Wild](https://mlsanity.com/paper/1903.03303/) (2019)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [Learn to Augment: Joint Data Augmentation and Network Optimization for Text Recognition](https://mlsanity.com/paper/2003.06606/) (2020)
- [Post-mortem Iris Recognition with Deep-Learning-based Image Segmentation](https://mlsanity.com/paper/1901.01708/) (2019)
- [Embedding Expansion: Augmentation in Embedding Space for Deep Metric Learning](https://mlsanity.com/paper/2003.02546/) (2020)
- [Iris Liveness Detection Competition (LivDet-Iris) -- The 2020 Edition](https://mlsanity.com/paper/2009.00749/) (2020)
- [D-NetPAD: An Explainable and Interpretable Iris Presentation Attack Detector](https://mlsanity.com/paper/2007.01381/) (2020)
- [SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation](https://mlsanity.com/paper/2006.04026/) (2020)
- [Domain-Specific Human-Inspired Binarized Statistical Image Features for Iris Recognition](https://mlsanity.com/paper/1807.05248/) (2018)
- [Visual Psychophysics for Making Face Recognition Algorithms More Explainable](https://mlsanity.com/paper/1803.07140/) (2018)
- [PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition](https://mlsanity.com/paper/1611.06448/) (2016)
- [Assessment of iris recognition reliability for eyes affected by ocular pathologies](https://mlsanity.com/paper/1809.00206/) (2018)
- [Iris Presentation Attack Detection: Where Are We Now?](https://mlsanity.com/paper/2006.13252/) (2020)
- [Performance of Humans in Iris Recognition: The Impact of Iris Condition and Annotation-driven Verification](https://mlsanity.com/paper/1807.05245/) (2018)
- [Deep Learning-Based Feature Extraction in Iris Recognition: Use Existing Models, Fine-tune or Train From Scratch?](https://mlsanity.com/paper/2002.08916/) (2020)

## Built on by (most cited first, 27 of 27)

- [CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning](https://mlsanity.com/paper/2112.00686/) (2021)
- [Deep Learning for Iris Recognition: A Survey](https://mlsanity.com/paper/2210.05866/) (2022)
- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [Iris Liveness Detection Competition (LivDet-Iris) -- The 2023 Edition](https://mlsanity.com/paper/2310.04541/) (2023)
- [ViSaRL: Visual Reinforcement Learning Guided by Human Saliency](https://mlsanity.com/paper/2403.10940/) (2024)
- [The Value of AI Guidance in Human Examination of Synthetically-Generated Faces](https://mlsanity.com/paper/2208.10544/) (2022)
- [Explain To Me: Salience-Based Explainability for Synthetic Face Detection Models](https://mlsanity.com/paper/2303.11969/) (2023)
- [Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Training](https://mlsanity.com/paper/2306.05527/) (2023)
- [MENTOR: Human Perception-Guided Pretraining for Increased Generalization](https://mlsanity.com/paper/2310.19545/) (2023)
- [Grains of Saliency: Optimizing Saliency-based Training of Biometric Attack Detection Models](https://mlsanity.com/paper/2405.00650/) (2024)
- [Taking Training Seriously: Human Guidance and Management-Based Regulation of Artificial Intelligence](https://mlsanity.com/paper/2402.08466/) (2024)
- [Mitigating Bias Using Model-Agnostic Data Attribution](https://mlsanity.com/paper/2405.05031/) (2024)
- [Improving generalization by mimicking the human visual diet](https://mlsanity.com/paper/2206.07802/) (2022)
- [Increasing Interpretability of Neural Networks By Approximating Human Visual Saliency](https://mlsanity.com/paper/2410.16115/) (2024)
- [Saliency-Guided Training for Fingerprint Presentation Attack Detection](https://mlsanity.com/paper/2505.02176/) (2025)
- [DiffGradCAM: A Class Activation Map Using the Full Model Decision to Solve Unaddressed Adversarial Attacks](https://mlsanity.com/paper/2506.08514/) (2025)
- [AutoSIGHT: Automatic Eye Tracking-based System for Immediate Grading of Human experTise](https://mlsanity.com/paper/2508.01015/) (2025)
- [Feature Chirality in Deep Learning Models](https://mlsanity.com/paper/2305.03966/) (2023)
- [Training Better Deep Learning Models Using Human Saliency](https://mlsanity.com/paper/2410.16190/) (2024)
- [Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue](https://mlsanity.com/paper/2412.05553/) (2024)
- [Almost Right: Making First-Layer Kernels Nearly Orthogonal Improves Model Generalization](https://mlsanity.com/paper/2504.16362/) (2025)
- [Divisive Decisions: Improving Salience-Based Training for Generalization in Binary Classification Tasks](https://mlsanity.com/paper/2507.17000/) (2025)
- [SAGE: Saliency-Guided Contrastive Embeddings](https://mlsanity.com/paper/2511.12744/) (2025)
- [Generalist Multimodal LLMs Gain Biometric Expertise via Human Salience](https://mlsanity.com/paper/2603.17173/) (2026)
- [VISER: Visually-Informed System for Enhanced Robustness in Iris Presentation Attack Detection](https://mlsanity.com/paper/2603.17859/) (2026)
- [Learn to Rank: Visual Attribution by Learning Importance Ranking](https://mlsanity.com/paper/2604.05819/) (2026)
- [What's Old is New Again: Classical Dimensionality Reduction for Efficient Saliency-Guided Biometric Attack Detection](https://mlsanity.com/paper/2606.13528/) (2026)

## Abstract (arXiv)

Deep learning has driven remarkable accuracy increases in many computer vision problems. One ongoing challenge is how to achieve the greatest accuracy in cases where training data is limited. A second ongoing challenge is that trained models oftentimes do not generalize well even to new data that is subjectively similar to the training set. We address these challenges in a novel way, with the first-ever (to our knowledge) exploration of encoding human judgement about salient regions of images into the training data. We compare the accuracy and generalization of a state-of-the-art deep learning algorithm for a difficult problem in biometric presentation attack detection when trained on (a) original images with typical data augmentations, and (b) the same original images transformed to encode human judgement about salient image regions. The latter approach results in models that achieve higher accuracy and better generalization, decreasing the error of the LivDet-Iris 2020 winner from 29.78% to 16.37%, and achieving impressive generalization in a leave-one-attack-type-out evaluation scenario. This work opens a new area of study for how to embed human intelligence into training strategies for deep learning to achieve high accuracy and generalization in cases of limited training data.