# Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing

arXiv: https://arxiv.org/abs/2407.04400 · HTML: https://mlsanity.com/paper/2407.04400/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022)
- [Non-convex Learning via Replica Exchange Stochastic Gradient MCMC](https://mlsanity.com/paper/2008.05367/) (2020)
- [LightNeuS: Neural Surface Reconstruction in Endoscopy using Illumination Decline](https://mlsanity.com/paper/2309.02777/) (2023)
- [REAL-Colon: A dataset for developing real-world AI applications in colonoscopy](https://mlsanity.com/paper/2403.02163/) (2024)
- [Few-Shot Anomaly Detection for Polyp Frames from Colonoscopy](https://mlsanity.com/paper/2006.14811/) (2020)
- [Infinite Feature Selection: A Graph-based Feature Filtering Approach](https://mlsanity.com/paper/2006.08184/) (2020)
- [Task-guided Domain Gap Reduction for Monocular Depth Prediction in Endoscopy](https://mlsanity.com/paper/2310.01663/) (2023)

## Built on by (most cited first, 3 of 3)

- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention](https://mlsanity.com/paper/2507.14560/) (2025)
- [Self-Attention And Beyond the Infinite: Towards Linear Transformers with Infinite Self-Attention](https://mlsanity.com/paper/2603.00175/) (2026)

## Abstract (arXiv)

To address overfitting and enhance model generalization in gastroenterological polyp size assessment, our study introduces Feature-Selection Gates (FSG) or Hard-Attention Gates (HAG) alongside Gradient Routing (GR) for dynamic feature selection. This technique aims to boost Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) by promoting sparse connectivity, thereby reducing overfitting and enhancing generalization. HAG achieves this through sparsification with learnable weights, serving as a regularization strategy. GR further refines this process by optimizing HAG parameters via dual forward passes, independently from the main model, to improve feature re-weighting. Our evaluation spanned multiple datasets, including CIFAR-100 for a broad impact assessment and specialized endoscopic datasets (REAL-Colon, Misawa, and SUN) focusing on polyp size estimation, covering over 200 polyps in more than 370,000 frames. The findings indicate that our HAG-enhanced networks substantially enhance performance in both binary and triclass classification tasks related to polyp sizing. Specifically, CNNs experienced an F1 Score improvement to 87.8% in binary classification, while in triclass classification, the ViT-T model reached an F1 Score of 76.5%, outperforming traditional CNNs and ViT-T models. To facilitate further research, we are releasing our codebase, which includes implementations for CNNs, multistream CNNs, ViT, and HAG-augmented variants. This resource aims to standardize the use of endoscopic datasets, providing public training-validation-testing splits for reliable and comparable research in gastroenterological polyp size estimation. The codebase is available at github.com/cosmoimd/feature-selection-gates.