mlsanityData to 7 Oct 2026

Wings: Learning Multimodal LLMs without Text-only Forgetting

arXiv 2406.03496 ↗First posted 5 Jun 2024cs.CL cs.AI cs.LGcited by 23

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs), initiated with a trained LLM, first align images with text and then fine-tune on multimodal mixed inputs. However, the MLLM catastrophically forgets the text-only instructions, which do not include images and can be addressed within the initial LLM. In this paper, we present Wings, a novel MLLM that excels in both text-only dialogues and multimodal comprehension. Analyzing MLLM attention in multimodal instructions reveals that text-only forgetting is related to the attention shifts from pre-image to post-image text. From that, we construct extra modules that act as the boosted learner to compensate for the attention shift. The complementary visual and textual learners, like "wings" on either side, are connected in parallel within each layer's attention block. Initially, image and text inputs are aligned with visual learners operating alongside the main attention, balancing focus on visual elements. Textual learners are later collaboratively integrated with attention-based routing to blend the outputs of the visual and textual learners. We design the Low-Rank Residual Attention (LoRRA) to guarantee high efficiency for learners. Our experimental results demonstrate that Wings outperforms equally-scaled MLLMs in both text-only and visual question-answering tasks. On a newly constructed Interleaved Image-Text (IIT) benchmark, Wings exhibits superior performance from text-only-rich to multimodal-rich question-answering tasks.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 123 of its 149 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

123 of 149 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models21 Dec
  9. 2023VILA: On Pre-training for Visual Language Models12 Dec
  10. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
8 more
  1. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  2. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 103

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
  13. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  14. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  15. 2023Qwen Technical Reportcited by 3,273
  16. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  17. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  18. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  19. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  20. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  21. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  22. 2020Learning to summarize from human feedbackcited by 2,562
  23. 2022Emergent Abilities of Large Language Modelscited by 2,483
  24. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  25. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  26. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  27. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  28. 2024Mixtral of Expertscited by 1,785
  29. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  30. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  31. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  32. 2023LIMA: Less Is More for Alignmentcited by 1,080
  33. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  34. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  35. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  36. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  37. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  38. 2021Florence: A New Foundation Model for Computer Visioncited by 802

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  2. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  3. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
  4. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  7. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
  8. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  9. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  10. 2024Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuningcited by 6
  11. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  12. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Modelscited by 5
11 more
  1. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3
  2. 2025LF2AR: Accounting for Layerwise Dynamics to Improve Multimodal Adaptation of Language Modelscited by 3
  3. 2026PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMscited by 3
  4. 2025LaVi: Efficient Large Vision-Language Models via Internal Feature Modulationcited by 2
  5. 2025A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMscited by 1
  6. 2025Instruction-Following Evaluation of Large Vision-Language Modelscited by 1
  7. 2026Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosiscited by 1
  8. 2024OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensionscited by 0
  9. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  10. 2026XPERT: Expert Knowledge Transfer for Effective Training of Language Modelscited by 0
  11. 2026NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  2. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
  3. 2025True Multimodal In-Context Learning Needs Attention to the Visual Context
  4. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
  5. 2025SAISA: Towards Multimodal Large Language Models with Both Training and Inference Efficiency
  6. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  7. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
  8. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction