mlsanityData to 7 Oct 2026

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

arXiv 2305.16103 ↗First posted 25 May 2023cs.CV cs.AI cs.CLcited by 64

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building general-purpose models that can perceive diverse real-world modalities and solve various tasks is an appealing target in artificial intelligence. In this paper, we present ChatBridge, a novel multimodal language model that leverages the expressive capabilities of language as the catalyst to bridge the gap between various modalities. We show that only language-paired two-modality data is sufficient to connect all modalities. ChatBridge leverages recent large language models (LLM) and extends their zero-shot capabilities to incorporate diverse multimodal inputs. ChatBridge undergoes a two-stage training. The first stage aligns each modality with language, which brings emergent multimodal correlation and collaboration abilities. The second stage instruction-finetunes ChatBridge to align it with user intent with our newly proposed multimodal instruction tuning dataset, named MULTIS, which covers a wide range of 16 multimodal tasks of text, image, video, and audio modalities. We show strong quantitative and qualitative results on zero-shot multimodal tasks covering text, image, video, and audio modalities. All codes, data, and models of ChatBridge will be open-sourced.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 58 of its 72 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

58 of 72 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  17. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  19. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  20. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  22. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  23. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  24. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  25. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  26. 2018nocaps: novel object captioning at scalecited by 444
  27. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  28. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  29. 2019Clotho: An Audio Captioning Datasetcited by 354
  30. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  31. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  32. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  33. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  34. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  35. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  36. 2019Audio-Visual Scene-Aware Dialogcited by 122
  37. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  38. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 26; 2025: 13; 2026: 8.

64 papers.

Sort:

By area:

  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  9. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  10. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  11. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  12. 2023Embodied Task Planning with Large Language Modelscited by 84
38 more
  1. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  2. 2023Audio-Visual LLM for Video Understandingcited by 76
  3. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  4. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  5. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  6. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  7. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  8. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  9. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  10. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  11. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  12. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  13. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  14. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  15. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  16. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  17. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  18. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  19. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  20. 2024Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseasescited by 14
  21. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Modelscited by 13
  22. 2024Model Composition for Multimodal Large Language Modelscited by 12
  23. 2024MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencingcited by 12
  24. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  25. 2024HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarkscited by 10
  26. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  27. 2024Slot-VLM: SlowFast Slots for Video-Language Modelingcited by 8
  28. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 7
  29. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  30. 2026AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognitioncited by 6
  31. 2025Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decouplingcited by 5
  32. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inferencecited by 5
  33. 2024Explore the Limits of Omni-modal Pretraining at Scalecited by 4
  34. 2024INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Modelcited by 4
  35. 2024BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidancecited by 4
  36. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  37. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  38. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Performance of Multimodal Language Models
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  5. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
  6. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
  7. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
  8. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond