# Pareto Self-Supervised Training for Few-Shot Learning

arXiv: https://arxiv.org/abs/2104.07841 · HTML: https://mlsanity.com/paper/2104.07841/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Revisiting Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1901.09005/) (2019)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution](https://mlsanity.com/paper/1804.09081/) (2018)
- [Boosting Few-Shot Visual Learning with Self-Supervision](https://mlsanity.com/paper/1906.05186/) (2019)
- [Few-Shot Image Recognition by Predicting Parameters from Activations](https://mlsanity.com/paper/1706.03466/) (2017)
- [Unsupervised Learning by Predicting Noise](https://mlsanity.com/paper/1704.05310/) (2017)
- [Predictive Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/1511.05467/) (2015)
- [When Does Self-supervision Improve Few-shot Learning?](https://mlsanity.com/paper/1910.03560/) (2019)
- [Diversity with Cooperation: Ensemble Methods for Few-Shot Classification](https://mlsanity.com/paper/1903.11341/) (2019)

## Built on by (most cited first, 30 of 52)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Prompt-based Distribution Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2312.09553/) (2023)
- [Rethinking Generalization in Few-Shot Classification](https://mlsanity.com/paper/2206.07267/) (2022)
- [Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization](https://mlsanity.com/paper/2210.02016/) (2022)
- [Few-Shot Classification with Contrastive Learning](https://mlsanity.com/paper/2209.08224/) (2022)
- [Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning](https://mlsanity.com/paper/2203.09064/) (2022)
- [Credit Card Fraud Detection Using Advanced Transformer Model](https://mlsanity.com/paper/2406.03733/) (2024)
- [Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models](https://mlsanity.com/paper/2410.05661/) (2024)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [Learning How to Propagate Messages in Graph Neural Networks](https://mlsanity.com/paper/2310.00697/) (2023)
- [Task Discrepancy Maximization for Fine-grained Few-Shot Classification](https://mlsanity.com/paper/2207.01376/) (2022)
- [Self-Supervision Can Be a Good Few-Shot Learner](https://mlsanity.com/paper/2207.09176/) (2022)
- [Identification of Prognostic Biomarkers for Stage III Non-Small Cell Lung Carcinoma in Female Nonsmokers Using Machine Learning](https://mlsanity.com/paper/2408.16068/) (2024)
- [Inferring Prototypes for Multi-Label Few-Shot Image Classification with Word Vector Guided Attention](https://mlsanity.com/paper/2112.01037/) (2021)
- [Hybrid Graph Neural Networks for Few-Shot Learning](https://mlsanity.com/paper/2112.06538/) (2021)
- [Label-Efficient Domain Generalization via Collaborative Exploration and Generalization](https://mlsanity.com/paper/2208.03644/) (2022)
- [Sparse Spatial Transformers for Few-Shot Learning](https://mlsanity.com/paper/2109.12932/) (2021)
- [Stochastic Multiple Target Sampling Gradient Descent](https://mlsanity.com/paper/2206.01934/) (2022)
- [Tree Structure-Aware Few-Shot Image Classification via Hierarchical Aggregation](https://mlsanity.com/paper/2207.06989/) (2022)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [Decoupled Self-supervised Learning for Non-Homophilous Graphs](https://mlsanity.com/paper/2206.03601/) (2022)
- [Advanced User Credit Risk Prediction Model using LightGBM, XGBoost and Tabnet with SMOTEENN](https://mlsanity.com/paper/2408.03497/) (2024)
- [Research on the Application of Computer Vision Based on Deep Learning in Autonomous Driving Technology](https://mlsanity.com/paper/2406.00490/) (2024)
- [GeNIe: Generative Hard Negative Images Through Diffusion](https://mlsanity.com/paper/2312.02548/) (2023)
- [Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integrated](https://mlsanity.com/paper/2406.04658/) (2024)
- [RankDNN: Learning to Rank for Few-shot Learning](https://mlsanity.com/paper/2211.15320/) (2022)
- [tSF: Transformer-based Semantic Filter for Few-Shot Learning](https://mlsanity.com/paper/2211.00868/) (2022)
- [Leveraging Invariant Principle for Heterophilic Graph Structure Distribution Shifts](https://mlsanity.com/paper/2408.09490/) (2024)
- [Multi-objective Deep Learning: Taxonomy and Survey of the State of the Art](https://mlsanity.com/paper/2412.01566/) (2024)
- [Multi-Scenario Combination Based on Multi-Agent Reinforcement Learning to Optimize the Advertising Recommendation System](https://mlsanity.com/paper/2407.02759/) (2024)

## Abstract (arXiv)

While few-shot learning (FSL) aims for rapid generalization to new concepts with little supervision, self-supervised learning (SSL) constructs supervisory signals directly computed from unlabeled data. Exploiting the complementarity of these two manners, few-shot auxiliary learning has recently drawn much attention to deal with few labeled data. Previous works benefit from sharing inductive bias between the main task (FSL) and auxiliary tasks (SSL), where the shared parameters of tasks are optimized by minimizing a linear combination of task losses. However, it is challenging to select a proper weight to balance tasks and reduce task conflict. To handle the problem as a whole, we propose a novel approach named as Pareto self-supervised training (PSST) for FSL. PSST explicitly decomposes the few-shot auxiliary problem into multiple constrained multi-objective subproblems with different trade-off preferences, and here a preference region in which the main task achieves the best performance is identified. Then, an effective preferred Pareto exploration is proposed to find a set of optimal solutions in such a preference region. Extensive experiments on several public benchmark datasets validate the effectiveness of our approach by achieving state-of-the-art performance.