# Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

arXiv: https://arxiv.org/abs/2203.09064 · HTML: https://mlsanity.com/paper/2203.09064/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 91 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 64)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)

## Built on by (most cited first, 27 of 27)

- [A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity](https://mlsanity.com/paper/2302.06015/) (2023)
- [SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification](https://mlsanity.com/paper/2211.16191/) (2022)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification](https://mlsanity.com/paper/2208.12398/) (2022)
- [Vision Transformers Need More Than Registers](https://mlsanity.com/paper/2602.22394/) (2026)
- [Towards Free Data Selection with General-Purpose Models](https://mlsanity.com/paper/2309.17342/) (2023)
- [Poisson Reweighted Laplacian Uncertainty Sampling for Graph-based Active Learning](https://mlsanity.com/paper/2210.15786/) (2022)
- [Mitigating the Effect of Incidental Correlations on Part-based Learning](https://mlsanity.com/paper/2310.00377/) (2023)
- [BECLR: Batch Enhanced Contrastive Few-Shot Learning](https://mlsanity.com/paper/2402.02444/) (2024)
- [MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Tasks](https://mlsanity.com/paper/2305.10309/) (2023)
- [Task-Oriented Channel Attention for Fine-Grained Few-Shot Classification](https://mlsanity.com/paper/2308.00093/) (2023)
- [Hierarchical Visual Categories Modeling: A Joint Representation Learning and Density Estimation Framework for Out-of-Distribution Detection](https://mlsanity.com/paper/2408.15580/) (2024)
- [PrototypeFormer: Learning to Explore Prototype Relationships for Few-shot Image Classification](https://mlsanity.com/paper/2310.03517/) (2023)
- [ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformer](https://mlsanity.com/paper/2402.16674/) (2024)
- [An Adaptive Plug-and-Play Network for Few-Shot Learning](https://mlsanity.com/paper/2302.09326/) (2023)
- [TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning](https://mlsanity.com/paper/2408.15566/) (2024)
- [Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning](https://mlsanity.com/paper/2405.03109/) (2024)
- [When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities](https://mlsanity.com/paper/2607.08605/) (2026)
- [Learning Primitive-aware Discriminative Representations for Few-shot Learning](https://mlsanity.com/paper/2208.09717/) (2022)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)
- [Plug-and-Play Feature Generation for Few-Shot Medical Image Classification](https://mlsanity.com/paper/2310.09471/) (2023)
- [ISWSST: Index-space-wave State Superposition Transformers for Multispectral Remotely Sensed Imagery Semantic Segmentation](https://mlsanity.com/paper/2407.03033/) (2024)
- [Enhancing Environmental Robustness in Few-shot Learning via Conditional Representation Learning](https://mlsanity.com/paper/2502.01183/) (2025)
- [Memory-Modular Classification: Learning to Generalize with Memory Replacement](https://mlsanity.com/paper/2504.06021/) (2025)
- [Shallow Deep Learning Can Still Excel in Fine-Grained Few-Shot Learning](https://mlsanity.com/paper/2507.22041/) (2025)
- [Color as the Impetus: Transforming Few-Shot Learner](https://mlsanity.com/paper/2507.22136/) (2025)
- [Semi-Supervised Hyperspectral Image Classification with Edge-Aware Superpixel Label Propagation and Adaptive Pseudo-Labeling](https://mlsanity.com/paper/2601.18049/) (2026)

## Abstract (arXiv)

This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a promising alternative to convolutional neural networks for visual recognition. But when there is no sufficient data, it gets stuck in overfitting and shows inferior performance. To improve data efficiency, we propose hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling and optimize the learnable parameters through latent attribute surrogates. The intrinsic image structure is utilized to reduce the ambiguity between foreground content and background noise by spectral tokens pooling. And the attribute surrogate learning scheme is designed to benefit from the rich visual information in image-label pairs instead of simple visual concepts assigned by their labels. Our Hierarchically Cascaded Transformers, called HCTransformers, is built upon a self-supervised learning framework DINO and is tested on several popular few-shot learning benchmarks. In the inductive setting, HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1-shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet, which demonstrates HCTransformers are efficient to extract discriminative features. Also, HCTransformers show clear advantages over SOTA few-shot classification methods in both 5-way 1-shot and 5-way 5-shot settings on four popular benchmark datasets, including miniImageNet, tieredImageNet, FC100, and CIFAR-FS. The trained weights and codes are available at https://github.com/StomachCold/HCTransformers.