mlsanityData to 7 Oct 2026

Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning

arXiv 2605.26110 ↗First posted 25 May 2026cs.LG cs.CL cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) achieve versatility by reformulating diverse tasks into a unified instruction-following framework via instruction tuning. However, real-world deployment requires continuous adaptation to emerging tasks, motivating Multimodal Continual Instruction Tuning (MCIT). Despite its growing importance, current MCIT research is hindered by severe engineering bottlenecks. Existing methods are typically implemented by directly modifying the base MLLM codebase, which imposes substantial implementation overhead and yields method-specific architectures that severely limit code reuse and fair comparison. To address this, we introduce Prism, a plug-in reproducible codebase specifically designed for scalable MCIT research. It separates algorithmic development from the backbone implementation via a lightweight plugin registration mechanism, enabling new strategies to be integrated as independent plugins without modifying the underlying MLLM codebase, thereby eliminating structural fragmentation and accelerating method development. Prism natively supports widely used large-scale training pipeline, thereby enabling reproducible and scalable MCIT experimentation. Code is available at https://github.com/LAMDA-CL/Prism.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 54 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

26 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning2 Feb
  2. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt8 Oct

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
12 more
  1. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  2. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  3. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  6. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  7. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  8. 2020FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understandingcited by 181
  9. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  10. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitaskscited by 19
  11. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  12. 2020ChemVA: Interactive Visual Analysis of Chemical Compound Similarity in Virtual Screeningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  2. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
  3. 2024SwitchCIT: Switching for Continual Instruction Tuning
  4. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
  5. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
  6. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
  7. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
  8. 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs