mlsanityData to 7 Oct 2026

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

arXiv 2303.16199 ↗First posted 28 Mar 2023cs.CV cs.AI cs.CLcited by 776

Not on the map. Why?

Cited by 263 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the word tokens at higher transformer layers. Then, a zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Besides language commands, our approach can be simply extended to multi-modal instructions for learning image-conditioned LLaMA model, which achieves superior reasoning performance on ScienceQA and COCO Caption benchmarks. Furthermore, we also evaluate the zero-initialized attention mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on traditional vision and language tasks, demonstrating the superior generalization capacity of our approach. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.

Key facts. Cited by 776 papers in the map (data through 7 Oct 2026). Builds on 89 of its 140 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

89 of 140 references.

References · 89

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  13. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  14. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  15. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  16. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  17. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  18. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  19. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  20. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  21. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  22. 2022Visual Prompt Tuningcited by 1,674
  23. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  24. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  25. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  26. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  27. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  28. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  29. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  30. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  31. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  32. 2021Grounded Language-Image Pre-trainingcited by 1,147
  33. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  34. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  35. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  36. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  37. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  38. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 217; 2024: 345; 2025: 164; 2026: 50.

776 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2023Instruction Tuning with GPT-4cited by 676
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  14. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  15. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  16. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  17. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  18. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  19. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  20. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  21. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  22. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  23. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  24. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  25. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  26. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  27. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  28. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  29. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  30. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  31. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  32. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  33. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  35. 2023Towards Generalist Biomedical AIcited by 286
  36. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  37. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  38. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276

50 most cited of 776 All 776 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  2. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Models
  3. 2026Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models
  4. 2025Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
  5. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimation
  6. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
  8. 2024Towards Effective and Efficient Continual Pre-training of Large Language Models