mlsanityData to 14 Oct 2026

Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data

arXiv 2112.01034 ↗First posted 2 Dec 2021cs.CV eess.IVcited by 5

Not on the map. Why?

Abstract (arXiv)

The human gaze is a cost-efficient physiological data that reveals human underlying attentional patterns. The selective attention mechanism helps the cognition system focus on task-relevant visual clues by ignoring the presence of distractors. Thanks to this ability, human beings can efficiently learn from a very limited number of training samples. Inspired by this mechanism, we aim to leverage gaze for medical image analysis tasks with small training data. Our proposed framework includes a backbone encoder and a Selective Attention Network (SAN) that simulates the underlying attention. The SAN implicitly encodes information such as suspicious regions that is relevant to the medical diagnose tasks by estimating the actual human gaze. Then we design a novel Auxiliary Attention Block (AAB) to allow information from SAN to be utilized by the backbone encoder to focus on selective areas. Specifically, this block uses a modified version of a multi-head attention layer to simulate the human visual search procedure. Note that the SAN and AAB can be plugged into different backbones, and the framework can be used for multiple medical image analysis tasks when equipped with task-specific heads. Our method is demonstrated to achieve superior performance on both 3D tumor segmentation and 2D chest X-ray classification tasks. We also show that the estimated gaze probability map of the SAN is consistent with an actual gaze fixation map obtained by board-certified doctors.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  6. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  7. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  8. 2018Group Normalizationcited by 1,524
  9. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  10. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  11. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  12. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
20 more
  1. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 511
  2. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  3. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  4. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  5. 2016The Importance of Skip Connections in Biomedical Image Segmentationcited by 126
  6. 2019Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognitioncited by 95
  7. 2020Gradient Centralization: A New Optimization Technique for Deep Neural Networkscited by 82
  8. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  9. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 69
  10. 2021Goal-Oriented Gaze Estimation for Zero-Shot Learningcited by 68
  11. 2020HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentationcited by 66
  12. 2018Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challengecited by 56
  13. 2020Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Developmentcited by 56
  14. 2019Mutual Context Network for Jointly Estimating Egocentric Gaze and Actionscited by 54
  15. 2017CNN-based Segmentation of Medical Imaging Datacited by 48
  16. 2020Integrating Human Gaze into Attention for Egocentric Activity Recognitioncited by 42
  17. 2018Object Referring in Videos with Language and Human Gazecited by 40
  18. 20193D Dilated Multi-Fiber Network for Real-time Brain Tumor Segmentation in MRIcited by 28
  19. 2019Multitask Learning to Improve Egocentric Action Recognitioncited by 23
  20. 20173D Convolutional Neural Networks for Brain Tumor Segmentation: A Comparison of Multi-resolution Architecturescited by 4

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 1.

5 papers.

  1. 2022Compound Prototype Matching for Few-shot Action Recognitioncited by 35
  2. 2022Probabilistic Integration of Object Level Annotations in Chest X-ray Classificationcited by 10
  3. 2024Weakly-supervised Medical Image Segmentation with Gaze Annotationscited by 10
  4. 2022Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learningcited by 6
  5. 2023Human Attention-Guided Explainable Artificial Intelligence for Computer Vision Modelscited by 6