mlsanityData to 14 Oct 2026

Vision Transformer attention alignment with human visual perception in aesthetic object evaluation

arXiv 2507.17616 ↗First posted 23 Jul 2025cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Visual attention mechanisms play a crucial role in human perception and aesthetic evaluation. Recent advances in Vision Transformers (ViTs) have demonstrated remarkable capabilities in computer vision tasks, yet their alignment with human visual attention patterns remains underexplored, particularly in aesthetic contexts. This study investigates the correlation between human visual attention and ViT attention mechanisms when evaluating handcrafted objects. We conducted an eye-tracking experiment with 30 participants (9 female, 21 male, mean age 24.6 years) who viewed 20 artisanal objects comprising basketry bags and ginger jars. Using a Pupil Labs eye-tracker, we recorded gaze patterns and generated heat maps representing human visual attention. Simultaneously, we analyzed the same objects using a pre-trained ViT model with DINO (Self-DIstillation with NO Labels), extracting attention maps from each of the 12 attention heads. We compared human and ViT attention distributions using Kullback-Leibler divergence across varying Gaussian parameters (sigma=0.1 to 3.0). Statistical analysis revealed optimal correlation at sigma=2.4 +-0.03, with attention head #12 showing the strongest alignment with human visual patterns. Significant differences were found between attention heads, with heads #7 and #9 demonstrating the greatest divergence from human attention (p< 0.05, Tukey HSD test). Results indicate that while ViTs exhibit more global attention patterns compared to human focal attention, certain attention heads can approximate human visual behavior, particularly for specific object features like buckles in basketry items. These findings suggest potential applications of ViT attention mechanisms in product design and aesthetic evaluation, while highlighting fundamental differences in attention strategies between human perception and current AI models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 83 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

13 of 83 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  3. 2021Transformers in Vision: A Surveycited by 1,013
  4. 2019Are Sixteen Heads Really Better than One?cited by 816
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  6. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  7. 2016What do different evaluation metrics tell us about saliency models?cited by 185
  8. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  9. 2023How Does Attention Work in Vision Transformers? A Visual Analytics Attemptcited by 30
  10. 2023Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attentioncited by 16
  11. 2020Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluationscited by 10
  12. 2014Supervised mid-level features for word image representationcited by 5
1 more
  1. 2024Emergence of Human-Like Attention in Self-Supervised Vision Transformers: an eye-tracking studycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Mapscited by 0
  2. 2026Cognitive Alignment At No Cost: Inducing Human Attention Biases For Interpretable Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers
  2. 2026Bridging Human and Model Attention: Explainability Analysis of CNN, Mamba, and ViT Architectures with Gaze-Based Validation
  3. 2025Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
  4. 2026Attention Alignment Between Humans and Vision-Language Models
  5. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
  6. 2026Interoceptive Divergence in Aesthetic Evaluation and Implications for Human-AI Alignment
  7. 2025Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests
  8. 2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessment