mlsanityData to 14 Oct 2026

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

arXiv 2605.14938 ↗First posted 14 May 2026cs.LG cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Continual learning in multimodal large language models (MLLMs) aims to sequentially acquire knowledge while mitigating catastrophic forgetting, yet existing methods face inherent limitations: architecture-based approaches incur additional computational overhead and often generalize poorly to new tasks, rehearsal-based methods rely on storing historical data, raising privacy and storage concerns, and conventional regularization-based strategies alone are insufficient to fully prevent parameter interference. We propose Octopus, a two-stage continual learning framework based on History-Free Gradient Orthogonalization (HiFGO), which enforces gradient-level orthogonality without historical task data. Our proposed two-stage finetuning strategy decouples task adaptation from regularization, achieving a principled balance between plasticity and stability. Experiments on UCIT show that Octopus establishes state-of-the-art performance, surpassing prior SOTA by 2.14% and 6.82% in terms of Avg and Last.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 72 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

55 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering1 Mar
  2. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models4 Nov
  3. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar

Other references · 52

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2016Learning without Forgettingcited by 2,496
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
38 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2022Visual Prompt Tuningcited by 1,681
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2017Continual Learning with Deep Generative Replaycited by 1,158
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2018Experience Replay for Continual Learningcited by 1,095
  10. 2021Learning to Prompt for Continual Learningcited by 944
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  12. 2023A Survey on Multimodal Large Language Modelscited by 909
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  14. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  16. 2017A Survey on Multi-Task Learningcited by 840
  17. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  18. 2019Large Scale Incremental Learningcited by 732
  19. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  20. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  21. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  22. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  23. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  24. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  25. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  26. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  27. 2023Class-Incremental Learning: A Surveycited by 209
  28. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  29. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  30. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  31. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  32. 2024Model merging with SVD to tie the Knotscited by 122
  33. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
  34. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  35. 2025ChartEdit: How Far Are MLLMs From Automating Chart Analysis? Evaluating MLLMs' Capability via Chart Editingcited by 32
  36. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Modelcited by 31
  37. 2024Adaptive Memory Replay for Continual Learningcited by 22
  38. 2024Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignmentcited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026BluePixel: Toward Native 4K Agentic Image Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
  2. 2024Continual Learning for Large Language Models: A Survey
  3. 2025Routing-Based Continual Learning for Multimodal Large Language Models
  4. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  5. 2025When Continue Learning Meets Multimodal Large Language Model: A Survey
  6. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
  7. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
  8. 2024Improving Multimodal Large Language Models Using Continual Learning