# MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

arXiv: https://arxiv.org/abs/2405.11446 · HTML: https://mlsanity.com/paper/2405.11446/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 19 May 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [Meta-Learning without Memorization](https://mlsanity.com/paper/1912.03820/) (2019)
- [Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features](https://mlsanity.com/paper/2203.03191/) (2022)
- [Boosting Natural Language Generation from Instructions with Meta-Learning](https://mlsanity.com/paper/2210.11617/) (2022)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)

## Built on by (most cited first, 10 of 10)

- [ContextNav: Towards Agentic Multimodal In-Context Learning](https://mlsanity.com/paper/2510.04560/) (2025)
- [ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation](https://mlsanity.com/paper/2508.04153/) (2025)
- [InstructRAG: Leveraging Retrieval-Augmented Generation on Instruction Graphs for LLM-Based Task Planning](https://mlsanity.com/paper/2504.13032/) (2025)
- [Meta-Learning at Scale for Large Language Models via Low-Rank Amortized Bayesian Meta-Learning](https://mlsanity.com/paper/2508.14285/) (2025)
- [Weak-Link Optimization for Multi-Agent Reasoning and Collaboration](https://mlsanity.com/paper/2604.15972/) (2026)
- [GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives](https://mlsanity.com/paper/2605.09027/) (2026)
- [Can Gradient Descent Simulate Prompting?](https://mlsanity.com/paper/2506.20989/) (2025)
- [Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning](https://mlsanity.com/paper/2602.12123/) (2026)
- [Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLM](https://mlsanity.com/paper/2605.01973/) (2026)
- [Meta-Learning Preferences for Multilingual LLM Alignment](https://mlsanity.com/paper/2607.13315/) (2026)

## Abstract (arXiv)

Adapting large language models (LLMs) to unseen tasks with in-context training samples without fine-tuning remains an important research problem. To learn a robust LLM that adapts well to unseen tasks, multiple meta-training approaches have been proposed such as MetaICL and MetaICT, which involve meta-training pre-trained LLMs on a wide variety of diverse tasks. These meta-training approaches essentially perform in-context multi-task fine-tuning and evaluate on a disjointed test set of tasks. Even though they achieve impressive performance, their goal is never to compute a truly general set of parameters. In this paper, we propose MAML-en-LLM, a novel method for meta-training LLMs, which can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We see an average increase of 2% on unseen domains in the performance while a massive 4% improvement on adaptation performance. Furthermore, we demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains by an average of 2%. Finally, we discuss the effects of type of tasks, optimizers and task complexity, an avenue barely explored in meta-training literature. Exhaustive experiments across 7 task settings along with two data settings demonstrate that models trained with MAML-en-LLM outperform SOTA meta-training approaches.