mlsanityData to 14 Oct 2026

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

arXiv 2304.15010 ↗First posted 28 Apr 2023cs.CV cs.AI cs.CLcited by 602

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.

Key facts. Cited by 602 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 78 references that are in the map. First posted on arXiv 28 Apr 2023.

Builds on

64 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  3. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
  4. 2022Visual Programming: Compositional visual reasoning without training18 Nov
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  13. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  14. 2022Visual Prompt Tuningcited by 1,681
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  17. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  18. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  19. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  20. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  21. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  22. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  23. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  25. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  26. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  27. 2023Instruction Tuning with GPT-4cited by 686
  28. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  29. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  30. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  31. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  32. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  33. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  34. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  35. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  36. 2023Tool Learning with Foundation Modelscited by 302
  37. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  38. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 176; 2024: 254; 2025: 143; 2026: 29.

602 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
38 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  4. 2023A Survey on Multimodal Large Language Modelscited by 909
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  6. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  8. 2023A Comprehensive Overview of Large Language Modelscited by 648
  9. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  12. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  13. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  14. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  15. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
  16. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  17. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  18. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  19. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  20. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  21. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  22. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  23. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  24. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  25. 2023Are aligned neural networks adversarially aligned?cited by 315
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  27. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  28. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  29. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  30. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  31. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  32. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  34. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  35. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  36. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  37. 2023Personalize Segment Anything Model with One Shotcited by 250
  38. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249

50 most cited of 602 All 602 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  2. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
  3. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
  4. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  5. 2023Improved Baselines with Visual Instruction Tuning
  6. 2026Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models
  7. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  8. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance