# Boosting Natural Language Generation from Instructions with Meta-Learning

arXiv: https://arxiv.org/abs/2210.11617 · HTML: https://mlsanity.com/paper/2210.11617/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Editable Neural Networks](https://mlsanity.com/paper/2004.00345/) (2020)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)
- [learn2learn: A Library for Meta-Learning Research](https://mlsanity.com/paper/2008.12284/) (2020)
- [Bayesian Hypernetworks](https://mlsanity.com/paper/1710.04759/) (2017)
- [Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems](https://mlsanity.com/paper/1905.05644/) (2019)
- [Hyperdecoders: Instance-specific decoders for multi-task NLP](https://mlsanity.com/paper/2203.08304/) (2022)
- [Meta-Transfer Learning for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2102.09397/) (2021)

## Built on by (most cited first, 15 of 15)

- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [Large Language Model Instruction Following: A Survey of Progresses and Challenges](https://mlsanity.com/paper/2303.10475/) (2023)
- [MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following](https://mlsanity.com/paper/2312.02436/) (2023)
- [HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisation](https://mlsanity.com/paper/2212.10315/) (2022)
- [HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts](https://mlsanity.com/paper/2402.12656/) (2024)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [AmbigNLG: Addressing Task Ambiguity in Instruction for NLG](https://mlsanity.com/paper/2402.17717/) (2024)
- [MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning](https://mlsanity.com/paper/2405.11446/) (2024)
- [InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning](https://mlsanity.com/paper/2504.13032/) (2025)
- [Investigating the Effectiveness of HyperTuning via Gisting](https://mlsanity.com/paper/2402.16817/) (2024)
- [From Instance Training to Instruction Learning: Task Adapters Generation from Instructions](https://mlsanity.com/paper/2406.12382/) (2024)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [From Trajectories to Instructions: Language-Conditioned Meta-Reinforcement Learning](https://mlsanity.com/paper/2607.18830/) (2026)
- [LoRA-generating hypernetworks for efficient on-device LLM generative personalization](https://mlsanity.com/paper/2609.24979/) (2026)

## Abstract (arXiv)

Recent work has shown that language models (LMs) trained with multi-task textitinstructional learning (MTIL) can solve diverse NLP tasks in zero- and few-shot settings with improved performance compared to prompt tuning. MTIL illustrates that LMs can extract and use information about the task from instructions beyond the surface patterns of the inputs and outputs. This suggests that meta-learning may further enhance the utilization of instructions for effective task transfer. In this paper we investigate whether meta-learning applied to MTIL can further improve generalization to unseen tasks in a zero-shot setting. Specifically, we propose to adapt meta-learning to MTIL in three directions: 1) Model Agnostic Meta Learning (MAML), 2) Hyper-Network (HNet) based adaptation to generate task specific parameters conditioned on instructions, and 3) an approach combining HNet and MAML. Through extensive experiments on the large scale Natural Instructions V2 dataset, we show that our proposed approaches significantly improve over strong baselines in zero-shot settings. In particular, meta-learning improves the effectiveness of instructions and is most impactful when the test tasks are strictly zero-shot (i.e. no similar tasks in the training set) and are "hard" for LMs, illustrating the potential of meta-learning for MTIL for out-of-distribution tasks.