mlsanityData to 7 Oct 2026

Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration

arXiv 2306.09093 ↗First posted 15 Jun 2023cs.CL cs.AI cs.CVcited by 168

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal LLMs and expand the capabilities of LLMs to handle diverse data modalities and address complex real-world scenarios.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 54 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

47 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
33 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  17. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  18. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  19. 2023The False Promise of Imitating Proprietary LLMscited by 254
  20. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  21. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  22. 2019Audio-Visual Scene-Aware Dialogcited by 122
  23. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  24. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  25. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  26. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  27. 2017Exploiting Cross-Sentence Context for Neural Machine Translationcited by 48
  28. 2021Multimodal Dialogue Response Generationcited by 32
  29. 2021A Survey on Multi-modal Summarizationcited by 26
  30. 2020Multi-Modal Open-Domain Dialoguecited by 19
  31. 2023A Paradigm Shift: The Future of Machine Translation Lies with Large Language Modelscited by 19
  32. 2023Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translationcited by 15
  33. 2016Much Ado About Time: Exhaustive Annotation of Temporal Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 84; 2025: 39; 2026: 12.

168 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2023A Survey on Multimodal Large Language Modelscited by 898
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  7. 2024Qwen2-Audio Technical Reportcited by 543
  8. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  9. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  10. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  11. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
38 more
  1. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  2. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  3. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  4. 2023Video Understanding with Large Language Models: A Surveycited by 203
  5. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  6. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  7. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  10. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  11. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  12. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  13. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  14. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  15. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  16. 2023Embodied Task Planning with Large Language Modelscited by 84
  17. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  18. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  19. 2023Audio-Visual LLM for Video Understandingcited by 76
  20. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  21. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  22. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  23. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  24. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  25. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  26. 2024Towards 3D Molecule-Text Interpretation in Language Modelscited by 65
  27. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  28. 2023DrugAssist: A Large Language Model for Molecule Optimizationcited by 57
  29. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  30. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  31. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  32. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  33. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  34. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  35. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
  36. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 45
  37. 2024Multimodal Table Understandingcited by 43
  38. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42

50 most cited of 168 All 168 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputs
  2. 2024A Review of Multi-Modal Large Language and Vision Models
  3. 2024ModaVerse: Efficiently Transforming Modalities with LLMs
  4. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  5. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  6. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMs
  7. 2024SoupLM: Model Integration in Large Language and Multi-Modal Models
  8. 2024The Revolution of Multimodal Large Language Models: A Survey