# Mask-guided Vision Transformer (MG-ViT) for Few-Shot Learning

arXiv: https://arxiv.org/abs/2205.09995 · HTML: https://mlsanity.com/paper/2205.09995/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 39 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Incorporating Convolution Designs into Visual Transformers](https://mlsanity.com/paper/2103.11816/) (2021)
- [LocalViT: Analyzing Locality in Vision Transformers](https://mlsanity.com/paper/2104.05707/) (2021)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Free Lunch for Few-shot Learning: Distribution Calibration](https://mlsanity.com/paper/2101.06395/) (2021)
- [Learning Active Learning from Data](https://mlsanity.com/paper/1703.03365/) (2017)
- [A Universal Representation Transformer Layer for Few-Shot Image Classification](https://mlsanity.com/paper/2006.11702/) (2020)
- [Rectifying the Shortcut Learning of Background for Few-Shot Learning](https://mlsanity.com/paper/2107.07746/) (2021)
- [Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning](https://mlsanity.com/paper/1702.08690/) (2017)
- [Rethinking Class Relations: Absolute-relative Supervised and Unsupervised Few-shot Learning](https://mlsanity.com/paper/2001.03919/) (2020)
- [Revisiting Fine-tuning for Few-shot Learning](https://mlsanity.com/paper/1910.00216/) (2019)
- [Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards](https://mlsanity.com/paper/1610.08120/) (2016)
- [Sparse Spatial Transformers for Few-Shot Learning](https://mlsanity.com/paper/2109.12932/) (2021)
- [Impact of base dataset design on few-shot image classification](https://mlsanity.com/paper/2007.08872/) (2020)

## Built on by (most cited first, 10 of 10)

- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [SAMAug: Point Prompt Augmentation for Segment Anything Model](https://mlsanity.com/paper/2307.01187/) (2023)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Mask-guided BERT for Few Shot Text Classification](https://mlsanity.com/paper/2302.10447/) (2023)
- [Rectify ViT Shortcut Learning by Visual Saliency](https://mlsanity.com/paper/2206.08567/) (2022)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition](https://mlsanity.com/paper/2210.00843/) (2022)
- [DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robot](https://mlsanity.com/paper/2310.12787/) (2023)
- [Hint-Aug: Drawing Hints from Foundation Vision Transformers Towards Boosted Few-Shot Parameter-Efficient Tuning](https://mlsanity.com/paper/2304.12520/) (2023)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)

## Abstract (arXiv)

Learning with little data is challenging but often inevitable in various application scenarios where the labeled data is limited and costly. Recently, few-shot learning (FSL) gained increasing attention because of its generalizability of prior knowledge to new tasks that contain only a few samples. However, for data-intensive models such as vision transformer (ViT), current fine-tuning based FSL approaches are inefficient in knowledge generalization and thus degenerate the downstream task performances. In this paper, we propose a novel mask-guided vision transformer (MG-ViT) to achieve an effective and efficient FSL on ViT model. The key idea is to apply a mask on image patches to screen out the task-irrelevant ones and to guide the ViT to focus on task-relevant and discriminative patches during FSL. Particularly, MG-ViT only introduces an additional mask operation and a residual connection, enabling the inheritance of parameters from pre-trained ViT without any other cost. To optimally select representative few-shot samples, we also include an active learning based sample selection method to further improve the generalizability of MG-ViT based FSL. We evaluate the proposed MG-ViT on both Agri-ImageNet classification task and ACFR apple detection task with gradient-weighted class activation mapping (Grad-CAM) as the mask. The experimental results show that the MG-ViT model significantly improves the performance when compared with general fine-tuning based ViT models, providing novel insights and a concrete approach towards generalizing data-intensive and large-scale deep learning models for FSL.