# LLaVA-c: Continual Improved Visual Instruction Tuning

arXiv: https://arxiv.org/abs/2506.08666 · HTML: https://mlsanity.com/paper/2506.08666/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [Learning Visual N-Grams from Web Data](https://mlsanity.com/paper/1612.09161/) (2016)
- [Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning](https://mlsanity.com/paper/2212.00259/) (2022)
- [CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model](https://mlsanity.com/paper/2403.08350/) (2024)

## Built on by (most cited first, 14 of 14)

- [Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training](https://mlsanity.com/paper/2507.05386/) (2025)
- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [Mind to Hand: Purposeful Robotic Control via Embodied Reasoning](https://mlsanity.com/paper/2512.08580/) (2025)
- [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2602.01990/) (2026)
- [Continual GUI Agents](https://mlsanity.com/paper/2601.20732/) (2026)
- [HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models](https://mlsanity.com/paper/2603.26362/) (2026)
- [MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook](https://mlsanity.com/paper/2509.14142/) (2025)
- [The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Models](https://mlsanity.com/paper/2604.04857/) (2026)
- [GUI-AC: Enhancing Continual Learning in GUI Agents](https://mlsanity.com/paper/2606.10522/) (2026)
- [HyperCLOVA X 32B Think](https://mlsanity.com/paper/2601.03286/) (2026)
- [CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2606.02502/) (2026)
- [ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2606.02576/) (2026)
- [Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2609.13742/) (2026)

## Abstract (arXiv)

Multimodal models like LLaVA-1.5 achieve state-of-the-art visual understanding through visual instruction tuning on multitask datasets, enabling strong instruction-following and multimodal performance. However, multitask learning faces challenges such as task balancing, requiring careful adjustment of data proportions, and expansion costs, where new tasks risk catastrophic forgetting and need costly retraining. Continual learning provides a promising alternative to acquiring new knowledge incrementally while preserving existing capabilities. However, current methods prioritize task-specific performance, neglecting base model degradation from overfitting to specific instructions, which undermines general capabilities. In this work, we propose a simple but effective method with two modifications on LLaVA-1.5: spectral-aware consolidation for improved task balance and unsupervised inquiry regularization to prevent base model degradation. We evaluate both general and task-specific performance across continual pretraining and fine-tuning. Experiments demonstrate that LLaVA-c consistently enhances standard benchmark performance and preserves general capabilities. For the first time, we show that task-by-task continual learning can achieve results that match or surpass multitask joint learning. The code will be publicly released.