# Complementary Roles of Activation and Parametric Memory in Few-Shot Learning

arXiv: https://arxiv.org/abs/2609.28250 · HTML: https://mlsanity.com/paper/2609.28250/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 25 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Large Language Models with Controllable Working Memory](https://mlsanity.com/paper/2211.05110/) (2022)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [Finding Skill Neurons in Pre-trained Transformer-based Language Models](https://mlsanity.com/paper/2211.07349/) (2022)
- [With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation](https://mlsanity.com/paper/2401.11504/) (2024)
- [Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics](https://mlsanity.com/paper/2505.00675/) (2025)
- [Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models](https://mlsanity.com/paper/2605.11887/) (2026)
- [Test-Time Training Provably Improves Transformers as In-context Learners](https://mlsanity.com/paper/2503.11842/) (2025)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models](https://mlsanity.com/paper/2601.04260/) (2026)
- [Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective](https://mlsanity.com/paper/2604.23267/) (2026)

## Abstract (arXiv)

At test time, large language models (LLMs) can encode historical information in activation memory (i.e., KV caches) and parametric memory (i.e., updated parameters). While activation memory is generally considered effective for factual recall and parametric memory for learning new tasks, their interplay remains unclear. In this work, we systematically investigate the role of memory in few-shot learning through controlled experiments. We find that activation memory is superior for recalling facts, whereas parametric memory does not consistently outperform activation memory in task learning. Moreover, our experiments show that the composite task, Conditional Arithmetic, requires the synergy of both memory types. Through neuron-level analysis, we find that the model activates distinct sets of neurons when accessing the same historical information through activation versus parametric memory. When both memory types are combined, the model recruits neurons from both sets, which is crucial for solving Conditional Arithmetic. These findings suggest that neither memory mechanism alone is sufficient for this composite task, highlighting the importance of their collaboration.