mlsanityData to 13 Oct 2026

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

arXiv 2312.11420 ↗First posted 18 Dec 2023cs.CL cs.AI cs.CVcited by 64

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper introduces an efficient strategy to transform Large Language Models (LLMs) into Multi-Modal Large Language Models (MLLMs). By conceptualizing this transformation as a domain adaptation process, i.e., transitioning from text understanding to embracing multiple modalities, we intriguingly note that, within each attention block, tuning LayerNorm suffices to yield strong performance. Moreover, when benchmarked against other tuning approaches like full parameter finetuning or LoRA, its benefits on efficiency are substantial. For example, when compared to LoRA on a 13B model scale, performance can be enhanced by an average of over 20% across five multi-modal tasks, and meanwhile, results in a significant reduction of trainable parameters by 41.9% and a decrease in GPU memory usage by 17.6%. On top of this LayerNorm strategy, we showcase that selectively tuning only with conversational data can improve efficiency further. Beyond these empirical outcomes, we provide a comprehensive analysis to explore the role of LayerNorm in adapting LLMs to the multi-modal domain and improving the expressive power of the model.

Key facts. Cited by 64 papers in the map (data through 13 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 44 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

37 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics13 Sep
  2. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets23 Aug
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2023Visual Instruction Tuningcited by 7,827
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
21 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  3. 2015VQA: Visual Question Answeringcited by 2,495
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  6. 2018Group Normalizationcited by 1,518
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,211
  9. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,193
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 967
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 868
  13. 2016Training Deep Nets with Sublinear Memory Costcited by 840
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 778
  15. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 435
  16. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  17. 2016Revisiting Batch Normalization For Practical Domain Adaptationcited by 286
  18. 2019Understanding and Improving Layer Normalizationcited by 213
  19. 2023One Wide Feedforward is All You Needcited by 15
  20. 2022AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modelingcited by 10
  21. 2023Vision Learners Meet Web Image-Text Pairscited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 24; 2025: 26; 2026: 14.

64 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 652
  2. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 50
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 44
  5. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 42
  6. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  9. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 16
  10. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  11. 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 13
  12. 2025ReDi: Rectified Discrete Flowcited by 13
38 more
  1. 2024ASPS: Augmented Segment Anything Model for Polyp Segmentationcited by 12
  2. 2024Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Modelscited by 11
  3. 2024textttBATCLIP: Bimodal Online Test-Time Adaptation for CLIPcited by 11
  4. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  5. 2024Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Modelscited by 10
  6. 2025Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflectioncited by 10
  7. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  8. 2025DeLoRA: Decoupling Angles and Strength in Low-rank Adaptationcited by 10
  9. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  10. 2024Understanding Why Large Language Models Can Be Ineffective in Time Series Analysis: The Impact of Modality Alignmentcited by 7
  11. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  12. 2025VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injectioncited by 5
  13. 2025PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmarkcited by 5
  14. 2024Efficient Open Set Single Image Test Time Adaptation of Vision Language Modelscited by 3
  15. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  16. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  17. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  18. 2025Segment Any Crack: Deep Semantic Segmentation Adaptation for Crack Detectioncited by 3
  19. 2025Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identificationcited by 3
  20. 2025Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classificationcited by 3
  21. 2024Lowering PyTorch's Memory Consumption for Selective Differentiationcited by 2
  22. 2024ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuningcited by 2
  23. 2025Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Modelscited by 2
  24. 2025SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networkscited by 2
  25. 2025Parameter Importance-Driven Continual Learning for Foundation Modelscited by 2
  26. 2025Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Taskscited by 2
  27. 2024Less is More: Fewer Interpretable Region via Submodular Subset Selectioncited by 1
  28. 2024CLIP's Visual Embedding Projector is a Few-shot Cornucopiacited by 1
  29. 2024Tuning Language Models by Mixture-of-Depths Ensemblecited by 1
  30. 2024NeST: Neighborhood-aware semantic alignment and temporal modulation for LLM based time series forecastingcited by 1
  31. 2025A Hessian-informed hyperparameter optimization for differential learning ratecited by 1
  32. 2025Adaptive parameter-efficient fine-tuning via Hessian-informed subset selectioncited by 1
  33. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  34. 2025Implementing Adaptations for Vision AutoRegressive Modelcited by 1
  35. 2025Simplifying Multi-Task Architectures Through Task-Specific Normalizationcited by 1
  36. 2026AtlasPatch: Scalable Foundation Model-based Tissue Detection and Patch Extraction for Computational Pathologycited by 1
  37. 2026Layer by layer, module by module: Choose both for optimal OOD probing of ViTcited by 1
  38. 2026Lightweight Cross-Spectral Face Recognition via Contrastive Alignment and Distillationcited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Outlier-weighed Layerwise Sampling for LLM Fine-tuning
  2. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  3. 2024TAIA: Large Language Models are Out-of-Distribution Data Learners
  4. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  5. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  6. 2024CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models
  7. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
  8. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning