Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning
On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) achieve versatility by reformulating diverse tasks into a unified instruction-following framework via instruction tuning. However, real-world deployment requires continuous adaptation to emerging tasks, motivating Multimodal Continual Instruction Tuning (MCIT). Despite its growing importance, current MCIT research is hindered by severe engineering bottlenecks. Existing methods are typically implemented by directly modifying the base MLLM codebase, which imposes substantial implementation overhead and yields method-specific architectures that severely limit code reuse and fair comparison. To address this, we introduce Prism, a plug-in reproducible codebase specifically designed for scalable MCIT research. It separates algorithmic development from the backbone implementation via a lightweight plugin registration mechanism, enabling new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, thereby eliminating structural fragmentation and accelerating method development. Prism natively supports widely used large-scale training pipeline, thereby enabling reproducible and scalable MCIT experimentation. Code is available at https://github.com/LAMDA-CL/Prism.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 54 references that are in the map. First posted on arXiv 25 May 2026.
Builds on
26 of 54 references.
Closest ancestors · 2
Same direction, newest first
- 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning2 Feb
- 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt8 Oct
Other references · 24
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2015VQA: Visual Question Answeringcited by 2,489
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2019Towards VQA Models That Can Readcited by 1,696
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
12 more
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
- 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
- 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
- 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
- 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
- 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
- 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 19
- 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
- 2020ChemVA: Interactive Visual Analysis of Chemical Compound Similarity in Virtual Screeningcited by 6
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuningcited by 0
- 2026ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuningcited by 0
- 2026Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuningcited by 0
- 2026Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning2 Aug 2026
- 2026ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning1 Jun 2026
- 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning1 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
- 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
- 2024SwitchCIT: Switching for Continual Instruction Tuning
- 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
- 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
- 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
- 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
- 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs