mlsanityData to 7 Oct 2026

PandaGPT: One Model To Instruction-Follow Them All

arXiv 2305.16355 ↗First posted 25 May 2023cs.CL cs.CVcited by 330

Not on the map. Why?

Cited by 154 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do. Our project page is at https://panda-gpt.github.io/.

Key facts. Cited by 330 papers in the map (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

22 of 37 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2020Learning to summarize from human feedbackcited by 2,562
10 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  5. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  6. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  7. 2022A Contrastive Framework for Neural Text Generationcited by 224
  8. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  9. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  10. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 87; 2024: 119; 2025: 81; 2026: 43.

330 papers.

Sort:

By area:

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  10. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  12. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
38 more
  1. 2024Qwen2-Audio Technical Reportcited by 543
  2. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  4. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  5. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  6. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  7. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  8. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  11. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  13. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  14. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  15. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  16. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  17. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  18. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 215
  19. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  20. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  21. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  22. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 177
  23. 2024An Introduction to Vision-Language Modelingcited by 175
  24. 2023AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Modelscited by 169
  25. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  26. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  27. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  28. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  29. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  30. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  31. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  32. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  33. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  34. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  35. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  36. 2024Large Language Models for Time Series: A Surveycited by 113
  37. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  38. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111

50 most cited of 330 All 330 (JSON)