# From Saliency to DINO: Saliency-guided Vision Transformer for Few-shot Keypoint Detection

arXiv: https://arxiv.org/abs/2304.03140 · HTML: https://mlsanity.com/paper/2304.03140/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation](https://mlsanity.com/paper/1907.12347/) (2019)

## Built on by (most cited first, 8 of 8)

- [OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection](https://mlsanity.com/paper/2409.19899/) (2024)
- [Meta-Point Learning and Refining for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2403.13647/) (2024)
- [SAM-helps-Shadow:When Segment Anything Model meet shadow removal](https://mlsanity.com/paper/2306.06113/) (2023)
- [Few-shot Shape Recognition by Learning Deep Shape-aware Features](https://mlsanity.com/paper/2312.01315/) (2023)
- [Recurrent Feature Mining and Keypoint Mixup Padding for Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2503.21140/) (2025)
- [Liveness Detection in Computer Vision: Transformer-based Self-Supervised Learning for Face Anti-Spoofing](https://mlsanity.com/paper/2406.13860/) (2024)
- [Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection](https://mlsanity.com/paper/2507.07994/) (2025)
- [Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks](https://mlsanity.com/paper/2512.04970/) (2025)

## Abstract (arXiv)

Unlike current deep keypoint detectors that are trained to recognize limited number of body parts, few-shot keypoint detection (FSKD) attempts to localize any keypoints, including novel or base keypoints, depending on the reference samples. FSKD requires the semantically meaningful relations for keypoint similarity learning to overcome the ubiquitous noise and ambiguous local patterns. One rescue comes with vision transformer (ViT) as it captures long-range relations well. However, ViT may model irrelevant features outside of the region of interest due to the global attention matrix, thus degrading similarity learning between support and query features. In this paper, we present a novel saliency-guided vision transformer, dubbed SalViT, for few-shot keypoint detection. Our SalViT enjoys a uniquely designed masked self-attention and a morphology learner, where the former introduces saliency map as a soft mask to constrain the self-attention on foregrounds, while the latter leverages the so-called power normalization to adjust morphology of saliency map, realizing ``dynamically changing receptive field''. Moreover, as salinecy detectors add computations, we show that attentive masks of DINO transformer can replace saliency. On top of SalViT, we also investigate i) transductive FSKD that enhances keypoint representations with unlabelled data and ii) FSKD under occlusions. We show that our model performs well on five public datasets and achieves ~10% PCK higher than the normally trained model under severe occlusions.