Accelerating Neural Self-Improvement via Bootstrapping
On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP
Abstract (arXiv)
Few-shot learning with sequence-processing neural networks (NNs) has recently attracted a new wave of attention in the context of large language models. In the standard N-way K-shot learning setting, an NN is explicitly optimised to learn to classify unlabelled inputs by observing a sequence of NK labelled examples. This pressures the NN to learn a learning algorithm that achieves optimal performance, given the limited number of training examples. Here we study an auxiliary loss that encourages further acceleration of few-shot learning, by applying recently proposed bootstrapped meta-learning to NN few-shot learners: we optimise the K-shot learner to match its own performance achievable by observing more than NK examples, using only NK examples. Promising results are obtained on the standard Mini-ImageNet dataset. Our code is public.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 26 of its 42 references that are in the map. First posted on arXiv 2 May 2023.
Builds on
26 of 42 references.
References · 26
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Rethinking Attention with Performerscited by 1,584
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
- 2022A Generalist Agentcited by 802
- 2022Transformers learn in-context by gradient descentcited by 714
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
14 more
- 2013Do Deep Nets Really Need to be Deep?cited by 466
- 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
- 2021Random Feature Attentioncited by 350
- 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
- 2017Meta Networkscited by 268
- 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
- 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
- 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
- 2022Transformers generalize differently from information stored in context vs in weightscited by 49
- 2021Bootstrapped Meta-Learningcited by 45
- 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
- 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
- 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 32
- 2022Meta-Learning with Self-Improving Momentum Targetcited by 9
Built on by
Citing papers per year: 2023: 1.
1 paper.
- 2023Metalearning Continual Learning Algorithmscited by 6
- 2023Metalearning Continual Learning Algorithms1 Dec 2023
Similar, not cited
Close in wording, no citation either way
- 2022Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning
- 2024Making Large Vision Language Models to be Good Few-shot Learners
- 2024What Makes Good Few-shot Examples for Vision-Language Models?
- 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning
- 2024Tiny models from tiny data: Textual and null-text inversion for few-shot distillation
- 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
- 2021Revisiting Self-Training for Few-Shot Learning of Language Model
- 2023Meta-training with Demonstration Retrieval for Efficient Few-shot Learning