# Using Human Perception to Regularize Transfer Learning

arXiv: https://arxiv.org/abs/2211.07885 · HTML: https://mlsanity.com/paper/2211.07885/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 89 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [A Survey of Optimization Methods from a Machine Learning Perspective](https://mlsanity.com/paper/1906.06821/) (2019)
- [The Implicit and Explicit Regularization Effects of Dropout](https://mlsanity.com/paper/2002.12915/) (2020)
- [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://mlsanity.com/paper/1904.01121/) (2019)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [CPTR: Full Transformer Network for Image Captioning](https://mlsanity.com/paper/2101.10804/) (2021)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [Do better ImageNet classifiers assess perceptual similarity better?](https://mlsanity.com/paper/2203.04946/) (2022)
- [CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning](https://mlsanity.com/paper/2112.00686/) (2021)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)

## Built on by (most cited first, 6 of 6)

- [Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Training](https://mlsanity.com/paper/2306.05527/) (2023)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency Response Scenarios](https://mlsanity.com/paper/2309.09518/) (2023)
- [MENTOR: Human Perception-Guided Pretraining for Increased Generalization](https://mlsanity.com/paper/2310.19545/) (2023)
- [Increasing Interpretability of Neural Networks By Approximating Human Visual Saliency](https://mlsanity.com/paper/2410.16115/) (2024)
- [Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue](https://mlsanity.com/paper/2412.05553/) (2024)

## Abstract (arXiv)

Recent trends in the machine learning community show that models with fidelity toward human perceptual measurements perform strongly on vision tasks. Likewise, human behavioral measurements have been used to regularize model performance. But can we transfer latent knowledge gained from this across different learning objectives? In this work, we introduce PERCEP-TL (Perceptual Transfer Learning), a methodology for improving transfer learning with the regularization power of psychophysical labels in models. We demonstrate which models are affected the most by perceptual transfer learning and find that models with high behavioral fidelity -- including vision transformers -- improve the most from this regularization by as much as 1.9% Top@1 accuracy points. These findings suggest that biologically inspired learning agents can benefit from human behavioral measurements as regularizers and psychophysical learned representations can be transferred to independent evaluation tasks.