# Tackling Distribution Shift in LLM via KILO: Knowledge-Instructed Learning for Continual Adaptation

arXiv: https://arxiv.org/abs/2508.03571 · HTML: https://mlsanity.com/paper/2508.03571/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Continual Learning of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2404.16789/) (2024)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Continual Pre-Training of Large Language Models: How to (re)warm your model?](https://mlsanity.com/paper/2308.04014/) (2023)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Continual Pre-Training Mitigates Forgetting in Language and Vision](https://mlsanity.com/paper/2205.09357/) (2022)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Continual Training of Language Models for Few-Shot Learning](https://mlsanity.com/paper/2210.05549/) (2022)
- [Error Sensitivity Modulation based Experience Replay: Mitigating Abrupt Representation Drift in Continual Learning](https://mlsanity.com/paper/2302.11344/) (2023)
- [An Investigation of Replay-based Approaches for Continual Learning](https://mlsanity.com/paper/2108.06758/) (2021)
- [Continual Learning on Graphs: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2402.11565/) (2024)
- [Continual Graph Learning: A Survey](https://mlsanity.com/paper/2301.12230/) (2023)
- [Recyclable Tuning for Continual Pre-training](https://mlsanity.com/paper/2305.08702/) (2023)
- [Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learning](https://mlsanity.com/paper/2501.01124/) (2025)
- [Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustness](https://mlsanity.com/paper/2407.12068/) (2024)
- [Graph Relation Aware Continual Learning](https://mlsanity.com/paper/2308.08259/) (2023)

## Built on by (most cited first, 1 of 1)

- [Linguistics and Human Brain: A Perspective of Computational Neuroscience](https://mlsanity.com/paper/2602.08275/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) often suffer from performance degradation when faced with domain shifts, primarily due to catastrophic forgetting. In this work, we propose KILO (Knowledge-Instructed Learning for Continual Adaptation), a novel continual learning framework that integrates dynamic knowledge graphs with instruction tuning. By leveraging retrieved domain-specific knowledge as guidance during training, KILO enhances both adaptability to new domains and retention of previously acquired knowledge. We pretrain our model on WikiText-103 and evaluate sequential adaptation across four diverse target domains: BioASQ, SciQ, TweetEval, and MIND. Our experiments demonstrate that KILO consistently outperforms strong baselines, including continual fine-tuning, ERNIE 2.0, and CPT, in terms of backward transfer, forward transfer, F1 score, retention rate, and training efficiency. These results highlight the effectiveness of combining structured knowledge retrieval and instruction prompting to overcome domain shift challenges in continual learning scenarios.