# Doc-to-LoRA: Learning to Instantly Internalize Contexts

arXiv: https://arxiv.org/abs/2602.15902 · HTML: https://mlsanity.com/paper/2602.15902/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 45 of its 58 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Personalization of Large Language Models: A Survey](https://mlsanity.com/paper/2411.00027/) (2024)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Reasoning Over Paragraph Effects in Situations](https://mlsanity.com/paper/1908.05852/) (2019)
- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)

## Built on by (most cited first, 30 of 43)

- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents](https://mlsanity.com/paper/2606.06087/) (2026)
- [Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents](https://mlsanity.com/paper/2606.16769/) (2026)
- [Training Transformers for KV Cache Compressibility](https://mlsanity.com/paper/2605.05971/) (2026)
- [Position: Weight Space Should Be a First-Class Generative AI Modality](https://mlsanity.com/paper/2605.18632/) (2026)
- [Context Distillation as Latent Memory Management](https://mlsanity.com/paper/2605.28889/) (2026)
- [Scaling Self-Evolving Agents via Parametric Memory](https://mlsanity.com/paper/2606.04536/) (2026)
- [Rethinking Continual Experience Internalization for Self-Evolving LLM Agents](https://mlsanity.com/paper/2606.04703/) (2026)
- [How LoRA Remembers? A Parametric Memory Law for LLM Finetuning](https://mlsanity.com/paper/2605.30260/) (2026)
- [Parametric Skills](https://mlsanity.com/paper/2606.30015/) (2026)
- [Sample-Efficient Learning from Agent Experience](https://mlsanity.com/paper/2607.21051/) (2026)
- [MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation](https://mlsanity.com/paper/2608.03275/) (2026)
- [The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptation](https://mlsanity.com/paper/2604.23750/) (2026)
- [Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution](https://mlsanity.com/paper/2606.06492/) (2026)
- [Doc-to-Atom: Learning to Compile and Compose Memory Atoms](https://mlsanity.com/paper/2606.12400/) (2026)
- [Cartridges++: KV Cache Compression without Off-Context Derailment](https://mlsanity.com/paper/2609.35621/) (2026)
- [Teaching agentic AI to generalize expert diagnostic reasoning in rare diseases](https://mlsanity.com/paper/2606.16149/) (2026)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [Towards Evolving Context Parameterization for Large Language Models](https://mlsanity.com/paper/2609.14168/) (2026)
- [RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents](https://mlsanity.com/paper/2609.23466/) (2026)
- [PatchKV: Weight-Space Compensation of KV Cache](https://mlsanity.com/paper/2609.39329/) (2026)
- [HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning](https://mlsanity.com/paper/2610.03039/) (2026)
- [Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms](https://mlsanity.com/paper/2605.08423/) (2026)
- [Good Agentic Friends Do Not Just Give Verbal Advice: They Can Update Your Weights](https://mlsanity.com/paper/2605.13839/) (2026)
- [Context Memorization for Efficient Long Context Generation](https://mlsanity.com/paper/2605.18226/) (2026)
- [Frames2LoRA: Parametric Video Internalization for Vision-Language Models](https://mlsanity.com/paper/2606.04351/) (2026)
- [Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models](https://mlsanity.com/paper/2606.06154/) (2026)
- [PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization](https://mlsanity.com/paper/2606.12942/) (2026)
- [Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapters](https://mlsanity.com/paper/2607.27594/) (2026)
- [Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models](https://mlsanity.com/paper/2608.09227/) (2026)

## Abstract (arXiv)

Long input sequences are central to in-context learning, document understanding, and multi-step reasoning of Large Language Models (LLMs). However, the quadratic attention cost of Transformers makes inference memory-intensive and slow. While context distillation (CD) can transfer information into model parameters, per-prompt distillation is impractical due to training costs and latency. To address these limitations, we propose Doc-to-LoRA (D2L), a lightweight hypernetwork that meta-learns to perform approximate CD within a single forward pass. Given an unseen prompt, D2L generates a LoRA adapter for a target LLM, enabling subsequent queries to be answered without re-consuming the original context, reducing latency and KV-cache memory consumption during inference of the target LLM. On a long-context needle-in-a-haystack task, D2L successfully learns to map contexts into adapters that store the needle information, achieving near-perfect zero-shot accuracy at sequence lengths exceeding the target LLM's native context window by more than 4x. On real-world QA datasets with limited compute, D2L outperforms standard CD while significantly reducing peak memory consumption and update latency. We envision that D2L can facilitate rapid adaptation of LLMs, opening up the possibility of frequent knowledge updates and personalized chat behavior.