# Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning

arXiv: https://arxiv.org/abs/2103.01315 · HTML: https://mlsanity.com/paper/2103.01315/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 83 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 30 of 62)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)

## Built on by (most cited first, 30 of 56)

- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Rectifying the Shortcut Learning of Background for Few-Shot Learning](https://mlsanity.com/paper/2107.07746/) (2021)
- [A Closer Look at Few-shot Classification Again](https://mlsanity.com/paper/2301.12246/) (2023)
- [OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2207.02261/) (2022)
- [Memorizing Complementation Network for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2208.05610/) (2022)
- [Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning](https://mlsanity.com/paper/2203.09064/) (2022)
- [Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing](https://mlsanity.com/paper/2501.04376/) (2025)
- [Matching Feature Sets for Few-Shot Image Classification](https://mlsanity.com/paper/2204.00949/) (2022)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [Channel Importance Matters in Few-Shot Image Classification](https://mlsanity.com/paper/2206.08126/) (2022)
- [Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroid](https://mlsanity.com/paper/2210.16834/) (2022)
- [EASY: Ensemble Augmented-Shot Y-shaped Learning: State-Of-The-Art Few-Shot Classification with Simple Ingredients](https://mlsanity.com/paper/2201.09699/) (2022)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning](https://mlsanity.com/paper/2110.09446/) (2021)
- [Complementary Calibration: Boosting General Continual Learning with Collaborative Distillation and Self-Supervision](https://mlsanity.com/paper/2109.02426/) (2021)
- [Label Hallucination for Few-Shot Classification](https://mlsanity.com/paper/2112.03340/) (2021)
- [Adaptive Dimension Reduction and Variational Inference for Transductive Few-Shot Classification](https://mlsanity.com/paper/2209.08527/) (2022)
- [Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning](https://mlsanity.com/paper/2510.11115/) (2025)
- [Task Affinity with Maximum Bipartite Matching in Few-Shot Learning](https://mlsanity.com/paper/2110.02399/) (2021)
- [Prototype Completion for Few-Shot Learning](https://mlsanity.com/paper/2108.05010/) (2021)
- [RankDNN: Learning to Rank for Few-shot Learning](https://mlsanity.com/paper/2211.15320/) (2022)
- [MHFC: Multi-Head Feature Collaboration for Few-Shot Learning](https://mlsanity.com/paper/2109.07785/) (2021)
- [tSF: Transformer-based Semantic Filter for Few-Shot Learning](https://mlsanity.com/paper/2211.00868/) (2022)
- [ESPT: A Self-Supervised Episodic Spatial Pretext Task for Improving Few-Shot Learning](https://mlsanity.com/paper/2304.13287/) (2023)
- [MDFM: Multi-Decision Fusing Model for Few-Shot Learning](https://mlsanity.com/paper/2112.00690/) (2021)
- [The Art of Camouflage: Few-Shot Learning for Animal Detection and Segmentation](https://mlsanity.com/paper/2304.07444/) (2023)
- [Exploring Category-correlated Feature for Few-shot Image Classification](https://mlsanity.com/paper/2112.07224/) (2021)
- [Continual Learning for Pose-Agnostic Object Recognition in 3D Point Clouds](https://mlsanity.com/paper/2209.04840/) (2022)
- [SpatialFormer: Semantic and Target Aware Attentions for Few-Shot Learning](https://mlsanity.com/paper/2303.09281/) (2023)

## Abstract (arXiv)

In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning (FSL) is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. FSL tasks have been predominantly solved by leveraging the ideas from gradient-based meta-learning and metric learning approaches. However, recent works have demonstrated the significance of powerful feature representations with a simple embedding network that can outperform existing sophisticated FSL algorithms. In this work, we build on this insight and propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations. Equivariance or invariance has been employed standalone in the previous works; however, to the best of our knowledge, they have not been used jointly. Simultaneous optimization for both of these contrasting objectives allows the model to jointly learn features that are not only independent of the input transformation but also the features that encode the structure of geometric transformations. These complementary sets of features help generalize well to novel classes with only a few data samples. We achieve additional improvements by incorporating a novel self-supervised distillation objective. Our extensive experimentation shows that even without knowledge distillation our proposed method can outperform current state-of-the-art FSL methods on five popular benchmark datasets.