# MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark

arXiv: https://arxiv.org/abs/2508.07307 · HTML: https://mlsanity.com/paper/2508.07307/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 60 references that are in the map.
- First posted on arXiv 10 Aug 2025.

## Builds on (most cited first, 30 of 51)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [RSVQA: Visual Question Answering for Remote Sensing Data](https://mlsanity.com/paper/2003.07333/) (2020)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [MapQA: A Dataset for Question Answering on Choropleth Maps](https://mlsanity.com/paper/2211.08545/) (2022)
- [PyCIL: A Python Toolbox for Class-Incremental Learning](https://mlsanity.com/paper/2112.12533/) (2021)
- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2411.15296/) (2024)

## Built on by (most cited first, 13 of 13)

- [Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance](https://mlsanity.com/paper/2511.15164/) (2025)
- [Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation](https://mlsanity.com/paper/2602.00722/) (2026)
- [Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2605.26110/) (2026)
- [CL-VISTA: Benchmarking Continual Learning in Video Large Language Models](https://mlsanity.com/paper/2604.00677/) (2026)
- [Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails](https://mlsanity.com/paper/2607.02020/) (2026)
- [KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls](https://mlsanity.com/paper/2510.19316/) (2025)
- [MAny: Merge Anything for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2604.14016/) (2026)
- [CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2606.02502/) (2026)
- [KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation](https://mlsanity.com/paper/2606.16256/) (2026)
- [Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs](https://mlsanity.com/paper/2606.20961/) (2026)
- [Progressive Multimodal Alignment for Continual Instruction Tuning](https://mlsanity.com/paper/2607.26947/) (2026)
- [PaLoRA: Paced Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2610.04226/) (2026)
- [Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2610.13215/) (2026)

## Abstract (arXiv)

Continual learning enables AI systems to acquire new knowledge while retaining previously learned information. While traditional unimodal methods have made progress, the rise of Multimodal Large Language Models (MLLMs) brings new challenges in Multimodal Continual Learning (MCL), where models are expected to address both catastrophic forgetting and cross-modal coordination. To advance research in this area, we present MCITlib, a comprehensive library for Multimodal Continual Instruction Tuning. MCITlib currently implements 8 representative algorithms and conducts evaluations on 3 benchmarks under 2 backbone models. The library will be continuously updated to support future developments in MCL. The codebase is released at https://github.com/Ghy0501/MCITlib.