# Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

arXiv: https://arxiv.org/abs/2411.05877 · HTML: https://mlsanity.com/paper/2411.05877/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 44 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](https://mlsanity.com/paper/2205.11388/) (2022)
- [Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning](https://mlsanity.com/paper/2404.00213/) (2024)
- [Instruction-tuned Language Models are Better Knowledge Learners](https://mlsanity.com/paper/2402.12847/) (2024)

## Built on by (most cited first, 26 of 26)

- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)
- [Training Plug-n-Play Knowledge Modules with Deep Context Distillation](https://mlsanity.com/paper/2503.08727/) (2025)
- [InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation](https://mlsanity.com/paper/2504.01707/) (2025)
- [LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents](https://mlsanity.com/paper/2606.06087/) (2026)
- [Training Transformers for KV Cache Compressibility](https://mlsanity.com/paper/2605.05971/) (2026)
- [Context Parametrization with Compositional Adapters](https://mlsanity.com/paper/2509.22158/) (2025)
- [ReasonCACHE: Teaching LLMs To Reason Without Weight Updates](https://mlsanity.com/paper/2602.02366/) (2026)
- [How LoRA Remembers? A Parametric Memory Law for LLM Finetuning](https://mlsanity.com/paper/2605.30260/) (2026)
- [Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution](https://mlsanity.com/paper/2606.06492/) (2026)
- [Doc-to-Atom: Learning to Compile and Compose Memory Atoms](https://mlsanity.com/paper/2606.12400/) (2026)
- [IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning](https://mlsanity.com/paper/2509.22621/) (2025)
- [Understanding Parametric Knowledge Injection in Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12668/) (2025)
- [Training Large Reasoning Models Efficiently via Progressive Thought Encoding](https://mlsanity.com/paper/2602.16839/) (2026)
- [Program-as-Weights: A Programming Paradigm for Fuzzy Functions](https://mlsanity.com/paper/2607.02512/) (2026)
- [PHASE-Tree: Modeling Character-State Evolution in Long-Horizon Role-Playing Dialogue](https://mlsanity.com/paper/2608.06975/) (2026)
- [RPMem: Learning Long-Term Recurrent Parametric Memory Across Sessions for LLM Agents](https://mlsanity.com/paper/2609.23466/) (2026)
- [Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference](https://mlsanity.com/paper/2605.26099/) (2026)
- [A Table Is Worth 64 Tokens: Pixel-level Compression for Multi-Table Document Question Answering](https://mlsanity.com/paper/2608.26949/) (2026)
- [Amortized Low-Rank Adaptation for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2609.12278/) (2026)
- [LoRA-generating hypernetworks for efficient on-device LLM generative personalization](https://mlsanity.com/paper/2609.24979/) (2026)
- [HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agents](https://mlsanity.com/paper/2609.30797/) (2026)
- [LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models](https://mlsanity.com/paper/2609.33268/) (2026)
- [Doc2LoRA Provides Decodable Representations of Scientific Ideas](https://mlsanity.com/paper/2609.38374/) (2026)
- [Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models](https://mlsanity.com/paper/2610.11715/) (2026)

## Abstract (arXiv)

Large language models (LMs) are typically adapted to improve performance on new contexts (eg text prompts that define new tasks or domains) through fine-tuning or prompting. However, there is an accuracy compute tradeoff -- fine-tuning incurs significant training cost and prompting increases inference overhead. We introduce GenerativeAdapter, an effective and efficient adaptation method that directly maps new contexts to low-rank LM adapters, thereby significantly reducing inference overhead with no need for finetuning. The adapter generator is trained via self-supervised learning, and can be used to adapt a single frozen LM for any new task simply by mapping the associated task or domain context to a new adapter. We apply GenerativeAdapter to two pretrained LMs (Mistral-7B-Instruct and Llama2-7B-Chat) and evaluate the adapted models in three adaption scenarios: knowledge acquisition from documents, learning from demonstrations, and personalization for users. In StreamingQA, our approach is effective in injecting knowledge into the LM's parameters, achieving a 63.5% improvement in F1 score over the model with supervised fine-tuning (from 19.5 to 31.5) for contexts as long as 32K tokens. In the MetaICL in-context learning evaluation, our method achieves an average accuracy of 44.9 across 26 tasks, outperforming the base model. On MSC, our method proves to be highly competitive in memorizing user information from conversations with a 4x reduction in computation and memory costs compared to prompting with full conversation history. Together, these results suggest that GenerativeAdapter should allow for general adaption to a wide range of different contexts.