# CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning

arXiv: https://arxiv.org/abs/2112.00686 · HTML: https://mlsanity.com/paper/2112.00686/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 1 Dec 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [CNN-generated images are surprisingly easy to spot... for now](https://mlsanity.com/paper/1912.11035/) (2019)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues](https://mlsanity.com/paper/2007.09355/) (2020)
- [Leveraging Frequency Analysis for Deep Fake Image Recognition](https://mlsanity.com/paper/2003.08685/) (2020)
- [On the Detection of Digital Face Manipulation](https://mlsanity.com/paper/1910.01717/) (2019)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [Video Face Manipulation Detection Through Ensemble of CNNs](https://mlsanity.com/paper/2004.07676/) (2020)
- [img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation](https://mlsanity.com/paper/2012.07791/) (2020)
- [Biometric Quality: Review and Application to Face Recognition with FaceQnet](https://mlsanity.com/paper/2006.03298/) (2020)
- [Human-Aided Saliency Maps Improve Generalization of Deep Learning](https://mlsanity.com/paper/2105.03492/) (2021)
- [Domain-Specific Human-Inspired Binarized Statistical Image Features for Iris Recognition](https://mlsanity.com/paper/1807.05248/) (2018)
- [Visual Psychophysics for Making Face Recognition Algorithms More Explainable](https://mlsanity.com/paper/1803.07140/) (2018)
- [Training CNNs in Presence of JPEG Compression: Multimedia Forensics vs Computer Vision](https://mlsanity.com/paper/2009.12088/) (2020)
- [A Study of the Human Perception of Synthetic Faces](https://mlsanity.com/paper/2111.04230/) (2021)
- [SREFI: Synthesis of Realistic Example Face Images](https://mlsanity.com/paper/1704.06693/) (2017)

## Built on by (most cited first, 30 of 33)

- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Deep Learning for Iris Recognition: A Survey](https://mlsanity.com/paper/2210.05866/) (2022)
- [Weakly-supervised deepfake localization in diffusion-generated images](https://mlsanity.com/paper/2311.04584/) (2023)
- [DeCLIP: Decoding CLIP representations for deepfake localization](https://mlsanity.com/paper/2409.08849/) (2024)
- [Iris Liveness Detection Competition (LivDet-Iris) -- The 2023 Edition](https://mlsanity.com/paper/2310.04541/) (2023)
- [ViSaRL: Visual Reinforcement Learning Guided by Human Saliency](https://mlsanity.com/paper/2403.10940/) (2024)
- [The Value of AI Guidance in Human Examination of Synthetically-Generated Faces](https://mlsanity.com/paper/2208.10544/) (2022)
- [Explain To Me: Salience-Based Explainability for Synthetic Face Detection Models](https://mlsanity.com/paper/2303.11969/) (2023)
- [Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Training](https://mlsanity.com/paper/2306.05527/) (2023)
- [MEGAN: Multi-Explanation Graph Attention Network](https://mlsanity.com/paper/2211.13236/) (2022)
- [Using Human Perception to Regularize Transfer Learning](https://mlsanity.com/paper/2211.07885/) (2022)
- [Do humans and Convolutional Neural Networks attend to similar areas during scene classification: Effects of task and image type](https://mlsanity.com/paper/2307.13345/) (2023)
- [NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency Response Scenarios](https://mlsanity.com/paper/2309.09518/) (2023)
- [MENTOR: Human Perception-Guided Pretraining for Increased Generalization](https://mlsanity.com/paper/2310.19545/) (2023)
- [Grains of Saliency: Optimizing Saliency-based Training of Biometric Attack Detection Models](https://mlsanity.com/paper/2405.00650/) (2024)
- [Taking Training Seriously: Human Guidance and Management-Based Regulation of Artificial Intelligence](https://mlsanity.com/paper/2402.08466/) (2024)
- [Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI](https://mlsanity.com/paper/2401.11835/) (2024)
- [Quantifying the Intrinsic Usefulness of Attributional Explanations for Graph Neural Networks with Artificial Simulatability Studies](https://mlsanity.com/paper/2305.15961/) (2023)
- [Analysis of Human Perception in Distinguishing Real and AI-Generated Faces: An Eye-Tracking Based Study](https://mlsanity.com/paper/2409.15498/) (2024)
- [Increasing Interpretability of Neural Networks By Approximating Human Visual Saliency](https://mlsanity.com/paper/2410.16115/) (2024)
- [Saliency-Guided Training for Fingerprint Presentation Attack Detection](https://mlsanity.com/paper/2505.02176/) (2025)
- [AutoSIGHT: Automatic Eye Tracking-based System for Immediate Grading of Human experTise](https://mlsanity.com/paper/2508.01015/) (2025)
- [Improving Model's Focus Improves Performance of Deep Learning-Based Synthetic Face Detectors](https://mlsanity.com/paper/2303.00818/) (2023)
- [Leveraging Human Salience to Improve Calorie Estimation](https://mlsanity.com/paper/2306.09527/) (2023)
- [A Unified Framework for Iris Anti-Spoofing: Introducing Iris Anti-Spoofing Cross-Domain-Testing Protocol and Masked-MoE Method](https://mlsanity.com/paper/2408.09752/) (2024)
- [Training Better Deep Learning Models Using Human Saliency](https://mlsanity.com/paper/2410.16190/) (2024)
- [Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue](https://mlsanity.com/paper/2412.05553/) (2024)
- [Almost Right: Making First-Layer Kernels Nearly Orthogonal Improves Model Generalization](https://mlsanity.com/paper/2504.16362/) (2025)
- [Divisive Decisions: Improving Salience-Based Training for Generalization in Binary Classification Tasks](https://mlsanity.com/paper/2507.17000/) (2025)
- [SAGE: Saliency-Guided Contrastive Embeddings](https://mlsanity.com/paper/2511.12744/) (2025)

## Abstract (arXiv)

Can deep learning models achieve greater generalization if their training is guided by reference to human perceptual abilities? And how can we implement this in a practical manner? This paper proposes a training strategy to ConveY Brain Oversight to Raise Generalization (CYBORG). This new approach incorporates human-annotated saliency maps into a loss function that guides the model's learning to focus on image regions that humans deem salient for the task. The Class Activation Mapping (CAM) mechanism is used to probe the model's current saliency in each training batch, juxtapose this model saliency with human saliency, and penalize large differences. Results on the task of synthetic face detection, selected to illustrate the effectiveness of the approach, show that CYBORG leads to significant improvement in accuracy on unseen samples consisting of face images generated from six Generative Adversarial Networks across multiple classification network architectures. We also show that scaling to even seven times the training data, or using non-human-saliency auxiliary information, such as segmentation masks, and standard loss cannot beat the performance of CYBORG-trained models. As a side effect of this work, we observe that the addition of explicit region annotation to the task of synthetic face detection increased human classification accuracy. This work opens a new area of research on how to incorporate human visual saliency into loss functions in practice. All data, code and pre-trained models used in this work are offered with this paper.