# Few-shot learning of neural networks from scratch by pseudo example optimization

arXiv: https://arxiv.org/abs/1802.03039 · HTML: https://mlsanity.com/paper/1802.03039/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2018.

## Built on by (most cited first, 15 of 15)

- [Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model](https://mlsanity.com/paper/2106.03310/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis](https://mlsanity.com/paper/2104.04868/) (2021)
- [Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2207.12106/) (2022)
- [SLaM: Student-Label Mixing for Distillation with Unlabeled Examples](https://mlsanity.com/paper/2302.03806/) (2023)
- [Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Data](https://mlsanity.com/paper/2101.06069/) (2021)
- [DistPro: Searching A Fast Knowledge Distillation Process via Meta Optimization](https://mlsanity.com/paper/2204.05547/) (2022)
- [Beyond Classification: Knowledge Distillation using Multi-Object Impressions](https://mlsanity.com/paper/2110.14215/) (2021)
- [Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systems](https://mlsanity.com/paper/2205.02604/) (2022)
- [Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation](https://mlsanity.com/paper/2102.07125/) (2021)
- [Confidence Conditioned Knowledge Distillation](https://mlsanity.com/paper/2107.06993/) (2021)
- [Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization](https://mlsanity.com/paper/2505.07675/) (2025)
- [Knowledge Distillation By Sparse Representation Matching](https://mlsanity.com/paper/2103.17012/) (2021)
- [Distillation from heterogeneous unlabeled collections](https://mlsanity.com/paper/2201.06507/) (2022)
- [Improving Diversity in Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2604.25795/) (2026)

## Abstract (arXiv)

In this paper, we propose a simple but effective method for training neural networks with a limited amount of training data. Our approach inherits the idea of knowledge distillation that transfers knowledge from a deep or wide reference model to a shallow or narrow target model. The proposed method employs this idea to mimic predictions of reference estimators that are more robust against overfitting than the network we want to train. Different from almost all the previous work for knowledge distillation that requires a large amount of labeled training data, the proposed method requires only a small amount of training data. Instead, we introduce pseudo training examples that are optimized as a part of model parameters. Experimental results for several benchmark datasets demonstrate that the proposed method outperformed all the other baselines, such as naive training of the target model and standard knowledge distillation.