mlsanityData to 7 Oct 2026

Using Human Perception to Regularize Transfer Learning

arXiv 2211.07885 ↗First posted 15 Nov 2022cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent trends in the machine learning community show that models with fidelity toward human perceptual measurements perform strongly on vision tasks. Likewise, human behavioral measurements have been used to regularize model performance. But can we transfer latent knowledge gained from this across different learning objectives? In this work, we introduce PERCEP-TL (Perceptual Transfer Learning), a methodology for improving transfer learning with the regularization power of psychophysical labels in models. We demonstrate which models are affected the most by perceptual transfer learning and find that models with high behavioral fidelity -- including vision transformers -- improve the most from this regularization by as much as 1.9% Top@1 accuracy points. These findings suggest that biologically inspired learning agents can benefit from human behavioral measurements as regularizers and psychophysical learned representations can be transferred to independent evaluation tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 41 of its 89 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

41 of 89 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
29 more
  1. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  3. 2020A Survey on Visual Transformercited by 653
  4. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  5. 2021Intriguing Properties of Vision Transformerscited by 446
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  7. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  8. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  9. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  10. 2019A Survey of Optimization Methods from a Machine Learning Perspectivecited by 90
  11. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  12. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  13. 2022Harmonizing the object recognition strategies of deep neural networks with humanscited by 59
  14. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  15. 2020Integrating Human Gaze into Attention for Egocentric Activity Recognitioncited by 42
  16. 2022Do better ImageNet classifiers assess perceptual similarity better?cited by 34
  17. 2021CYBORG: Blending Human Saliency Into the Loss Improves Deep Learningcited by 33
  18. 2018AGIL: Learning Attention from Human for Visuomotor Taskscited by 32
  19. 2018Deep Predictive Coding Network for Object Recognitioncited by 23
  20. 2018Visual Psychophysics for Making Face Recognition Algorithms More Explainablecited by 19
  21. 2016PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognitioncited by 18
  22. 2020Machine versus Human Attention in Deep Reinforcement Learning Taskscited by 16
  23. 2021Environment Predictive Coding for Embodied Agentscited by 12
  24. 2019Measuring Human Perception to Improve Handwritten Document Transcriptioncited by 11
  25. 2022Measuring Human Perception to Improve Open Set Recognitioncited by 7
  26. 2018A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perceptioncited by 6
  27. 2016Predicting First Impressions with Deep Learningcited by 4
  28. 2018A Neurobiological Evaluation Metric for Neural Network Model Searchcited by 3
  29. 2022Guiding Machine Perception with Psychophysicscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023Teaching AI to Teach: Leveraging Limited Human Salience Data Into Unlimited Saliency-Based Trainingcited by 8
  2. 2023Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproductscited by 4
  3. 2023NOMAD: A Natural, Occluded, Multi-scale Aerial Dataset, for Emergency Response Scenarioscited by 4
  4. 2023MENTOR: Human Perception-Guided Pretraining for Increased Generalizationcited by 4
  5. 2024Increasing Interpretability of Neural Networks By Approximating Human Visual Saliencycited by 1
  6. 2024Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescuecited by 0