# Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition

arXiv: https://arxiv.org/abs/2110.12183 · HTML: https://mlsanity.com/paper/2110.12183/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 69 references that are in the map.
- First posted on arXiv 23 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Recent Advances in Convolutional Neural Networks](https://mlsanity.com/paper/1512.07108/) (2015)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up](https://mlsanity.com/paper/1903.02827/) (2019)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [Deep Contextual Attention for Human-Object Interaction Detection](https://mlsanity.com/paper/1910.07721/) (2019)
- [Contextual Action Recognition with R*CNN](https://mlsanity.com/paper/1505.01197/) (2015)
- [Real-time Distracted Driver Posture Classification](https://mlsanity.com/paper/1706.09498/) (2017)
- [Driver Distraction Identification with an Ensemble of Convolutional Neural Networks](https://mlsanity.com/paper/1901.09097/) (2019)
- [DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment](https://mlsanity.com/paper/1606.05675/) (2016)
- [Wide-Slice Residual Networks for Food Recognition](https://mlsanity.com/paper/1612.06543/) (2016)
- [Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering](https://mlsanity.com/paper/1604.04808/) (2016)
- [Single Image Action Recognition using Semantic Body Part Actions](https://mlsanity.com/paper/1612.04520/) (2016)
- [Actions and Attributes from Wholes and Parts](https://mlsanity.com/paper/1412.2604/) (2014)
- [FoodNet: Recognizing Foods Using Ensemble of Deep Networks](https://mlsanity.com/paper/1709.09429/) (2017)
- [Action Classification via Concepts and Attributes](https://mlsanity.com/paper/1605.07824/) (2016)
- [Loss Guided Activation for Action Recognition in Still Images](https://mlsanity.com/paper/1812.04194/) (2018)

## Built on by (most cited first, 9 of 9)

- [SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization](https://mlsanity.com/paper/2209.02109/) (2022)
- [Fine-Grained Sports, Yoga, and Dance Postures Recognition: A Benchmark Analysis](https://mlsanity.com/paper/2308.00323/) (2023)
- [RAFA-Net: Region Attention Network For Food Items And Agricultural Stress Recognition](https://mlsanity.com/paper/2410.12718/) (2024)
- [PND-Net: Plant Nutrition Deficiency and Disease Classification using Graph Convolutional Network](https://mlsanity.com/paper/2410.12742/) (2024)
- [Deep Neural Networks Fused with Textures for Image Classification](https://mlsanity.com/paper/2308.01813/) (2023)
- [Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision](https://mlsanity.com/paper/2310.05202/) (2023)
- [Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning](https://mlsanity.com/paper/2411.13181/) (2024)
- [MaskAttn-UNet: A Mask Attention-Driven Framework for Universal Low-Resolution Image Segmentation](https://mlsanity.com/paper/2503.10686/) (2025)
- [Women Sport Actions Dataset for Visual Classification Using Small Scale Training Data](https://mlsanity.com/paper/2507.10969/) (2025)

## Abstract (arXiv)

This paper presents a novel keypoints-based attention mechanism for visual recognition in still images. Deep Convolutional Neural Networks (CNNs) for recognizing images with distinctive classes have shown great success, but their performance in discriminating fine-grained changes is not at the same level. We address this by proposing an end-to-end CNN model, which learns meaningful features linking fine-grained changes using our novel attention mechanism. It captures the spatial structures in images by identifying semantic regions (SRs) and their spatial distributions, and is proved to be the key to modelling subtle changes in images. We automatically identify these SRs by grouping the detected keypoints in a given image. The ``usefulness'' of these SRs for image recognition is measured using our innovative attentional mechanism focusing on parts of the image that are most relevant to a given task. This framework applies to traditional and fine-grained image recognition tasks and does not require manually annotated regions (e.g. bounding-box of body parts, objects, etc.) for learning and prediction. Moreover, the proposed keypoints-driven attention mechanism can be easily integrated into the existing CNN models. The framework is evaluated on six diverse benchmark datasets. The model outperforms the state-of-the-art approaches by a considerable margin using Distracted Driver V1 (Acc: 3.39%), Distracted Driver V2 (Acc: 6.58%), Stanford-40 Actions (mAP: 2.15%), People Playing Musical Instruments (mAP: 16.05%), Food-101 (Acc: 6.30%) and Caltech-256 (Acc: 2.59%) datasets.