# On the Power of Context-Enhanced Learning in LLMs

arXiv: https://arxiv.org/abs/2503.01821 · HTML: https://mlsanity.com/paper/2503.01821/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2025.

## Built on by (most cited first, 5 of 5)

- [Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning](https://mlsanity.com/paper/2601.20829/) (2026)
- [Fine-Tuned In-Context Learners for Efficient Adaptation](https://mlsanity.com/paper/2512.19879/) (2025)
- [URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training](https://mlsanity.com/paper/2505.16570/) (2025)
- [IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning](https://mlsanity.com/paper/2509.22621/) (2025)
- [Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining](https://mlsanity.com/paper/2511.21613/) (2025)

## Abstract (arXiv)

We formalize a new concept for LLMs, context-enhanced learning. It involves standard gradient-based learning on text except that the context is enhanced with additional data on which no auto-regressive gradients are computed. This setting is a gradient-based analog of usual in-context learning (ICL) and appears in some recent works. Using a multi-step reasoning task, we prove in a simplified setting that context-enhanced learning can be exponentially more sample-efficient than standard learning when the model is capable of ICL. At a mechanistic level, we find that the benefit of context-enhancement arises from a more accurate gradient learning signal. We also experimentally demonstrate that it appears hard to detect or recover learning materials that were used in the context during training. This may have implications for data security as well as copyright.