mlsanityData to 14 Oct 2026

Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

arXiv 2406.11354 ↗First posted 17 Jun 2024cs.CL cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. Moreover, for Multimodal Large Language Models (MLLMs) which are composed of the LLM base and visual projector (e.g. LLaVA), a significant decline in performance on language benchmarks was observed compared to their single-modality counterparts. To address these challenges, we introduce a novel model-agnostic self-decompression method, Tree Generation (TG), that decompresses knowledge within LLMs into the training corpus. This paper focuses on TG-SFT, which can synthetically generate SFT data for the instruction tuning steps. By incorporating the dumped corpus during SFT for MLLMs, we significantly reduce the forgetting problem.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

38 of 57 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
26 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  10. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  11. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  12. 2023Llemma: An Open Language Model For Mathematicscited by 388
  13. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  14. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensingcited by 328
  15. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  16. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  17. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  18. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  19. 2023LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugincited by 82
  20. 2022An Introduction to Neural Data Compressioncited by 79
  21. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  22. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  23. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 40
  24. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  25. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  26. 2024Towards Optimal Learning of Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Talking to Yourself: Defying Forgetting in Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-training Large Language Models through Knowledge Detection
  2. 2025Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
  3. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models
  4. 2025A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios
  5. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Models
  6. 2024Improving Multimodal Large Language Models Using Continual Learning
  7. 2025Context-Free Synthetic Data Mitigates Forgetting
  8. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models