OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Recent advances in multimodal large language models (LLMs) have led to significant progress in understanding, generation, and retrieval tasks. However, current solutions often treat these tasks in isolation or require training LLMs from scratch, resulting in high computational costs and limited generalization across modalities. In this work, we present OmniBridge, a unified and modular multimodal framework that supports vision-language understanding, generation, and retrieval within a unified architecture. OmniBridge adopts a language-centric design that reuses pretrained LLMs and introduces a lightweight bidirectional latent alignment module. To address the challenge of task interference, we propose a two-stage decoupled training strategy: supervised fine-tuning and latent space alignment for aligning LLM behavior with multimodal reasoning, and semantic-guided diffusion training to align cross-modal latent spaces via learnable query embeddings. Extensive experiments across a wide range of benchmarks demonstrate that OmniBridge achieves competitive or state-of-the-art performance in all three tasks. Moreover, our results highlight the effectiveness of latent space alignment for unifying multimodal modeling under a shared representation space. Code and models are released at https://github.com/xiao-xt/OmniBridge.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 114 of its 126 references that are in the map. First posted on arXiv 23 Sep 2025.
Builds on
114 of 126 references.
Closest ancestors · 19
Same direction, newest first
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
- 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?7 Oct
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
- 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
- 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
- 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
- 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
- 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
- 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
7 more
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
- 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
- 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
- 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
- 2023A Survey on Multimodal Large Language Models23 Jun
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
Other references · 95
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Visual Instruction Tuningcited by 7,869
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2022Video Diffusion Modelscited by 2,305
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
- 2019Towards VQA Models That Can Readcited by 1,718
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
- 2016A Diagram Is Worth A Dozen Imagescited by 804
- 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
- 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
50 most cited of 95 All 95 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 4.
5 papers.
- 2026Lance: Unified Multimodal Modeling by Multi-Task Synergycited by 12
- 2025GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Modelscited by 6
- 2026Self-Corrected Image Generation with Explainable Latent Rewardscited by 1
- 2026LatentUMM: Dual Latent Alignment for Unified Multimodal Modelscited by 1
- 2026IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generationcited by 0
- 2026IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation8 Jun 2026
- 2026Lance: Unified Multimodal Modeling by Multi-Task Synergy18 May 2026
- 2026LatentUMM: Dual Latent Alignment for Unified Multimodal Models18 May 2026
- 2026Self-Corrected Image Generation with Explainable Latent Rewards26 Mar 2026
- 2025GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models14 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2025Scaling Language-Centric Omnimodal Representation Learning
- 2024OmniBench: Towards The Future of Universal Omni-Language Models
- 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
- 2024A dual contrastive framework
- 2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
- 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
- 2026Context Unrolling in Omni Models