# Knowledge Interaction in Hypernetwork-based Continual Learning: Conflict and Composition

arXiv: https://arxiv.org/abs/2610.15677 · HTML: https://mlsanity.com/paper/2610.15677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 11 Oct 2026.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Entity-Based Knowledge Conflicts in Question Answering](https://mlsanity.com/paper/2109.05052/) (2021)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Simple and Scalable Strategies to Continually Pre-train Large Language Models](https://mlsanity.com/paper/2403.08763/) (2024)
- [Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal](https://mlsanity.com/paper/2403.01244/) (2024)
- [LoRA vs Full Fine-tuning: An Illusion of Equivalence](https://mlsanity.com/paper/2410.21228/) (2024)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2402.18154/) (2024)
- [Text-to-LoRA: Instant Transformer Adaption](https://mlsanity.com/paper/2506.06105/) (2025)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Taming Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2503.10996/) (2025)
- [Context versus Prior Knowledge in Language Models](https://mlsanity.com/paper/2404.04633/) (2024)
- [SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass](https://mlsanity.com/paper/2602.06358/) (2026)

## Abstract (arXiv)

Continual learning aims to adapt pretrained large language models (LLMs) to evolving data while preserving their pretrained capabilities. A central challenge is catastrophic forgetting: adaptation to new data can impair performance on previously learned knowledge. Since continual learning commonly changes model parameters, the model's predictions may draw on knowledge from the base model, the parameter update, and the input context. Understanding how these sources interact is essential: conflicts reveal whether new knowledge can override outdated knowledge, while multi-hop reasoning tests whether new knowledge can be composed with existing knowledge to support generalization. We study these interactions in a hypernetwork-based adaptation framework, where a shared network generates adapters from context, allowing a frozen base model to adapt without task-specific updates. We find that, under knowledge conflict, models preferentially follow the context, and their choices depend more on where knowledge is presented than on its content. We also show that context and adapter knowledge perturb the base model's representations along directions orthogonal to its clean hidden states, enabling us to derive steering vectors that control which source the model follows. In multi-hop reasoning, we find that models fail to reliably compose knowledge across the three sources, with the key breakdown arising from the interaction of the hypernetwork with both the base model and the context. Code and reproduction materials: https://github.com/missswiftie/Knowledge-Interaction-in-CL