mlsanityData to 7 Oct 2026

CROME: Cross-Modal Adapters for Efficient Multimodal LLM

arXiv 2408.06610 ↗First posted 13 Aug 2024cs.CV cs.CL cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) demonstrate remarkable image-language capabilities, but their widespread use faces challenges in cost-effective training and adaptation. Existing approaches often necessitate expensive language model retraining and limited adaptability. Additionally, the current focus on zero-shot performance improvements offers insufficient guidance for task-specific tuning. We propose CROME, an efficient vision-language instruction tuning framework. It features a novel gated cross-modal adapter that effectively combines visual and textual representations prior to input into a frozen LLM. This lightweight adapter, trained with minimal parameters, enables efficient cross-modal understanding. Notably, CROME demonstrates superior zero-shot performance on standard visual question answering and instruction-following benchmarks. Moreover, it yields fine-tuning with exceptional parameter efficiency, competing with task-specific specialist state-of-the-art methods. CROME demonstrates the potential of pre-LM alignment for building scalable, adaptable, and parameter-efficient multimodal models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 58 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

54 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
34 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2016A Diagram Is Worth A Dozen Imagescited by 793
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  14. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  16. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  17. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  18. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  19. 2016Language Modeling with Gated Convolutional Networkscited by 688
  20. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  21. 2017Learning multiple visual domains with residual adapterscited by 553
  22. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  23. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  24. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  25. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  26. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  27. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  28. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  29. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  30. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  31. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  32. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  33. 2023Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Trainingcited by 21
  34. 2023PILL: Plug Into LLM with Adapter Expert and Attention Gatecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025Global Minimizers of Sigmoid Contrastive Losscited by 4
  3. 2026Conditional Flow Matching for Visually-Guided Acoustic Highlightingcited by 1
  4. 2026VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Performance of Multimodal Language Models
  2. 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
  3. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  4. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  5. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMs
  6. 2023Improved Baselines with Visual Instruction Tuning
  7. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  8. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models