mlsanityData to 14 Oct 2026

OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation

arXiv 2505.23661 ↗First posted 29 May 2025cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

In this report, we present OpenUni, a simple, lightweight, and fully open-source baseline for unifying multimodal understanding and generation. Inspired by prevailing practices in unified model learning, we adopt an efficient training strategy that minimizes the training complexity and overhead by bridging the off-the-shelf multimodal large language models (LLMs) and diffusion models through a set of learnable queries and a light-weight transformer-based connector. With a minimalist choice of architecture, we demonstrate that OpenUni can: 1) generate high-quality and instruction-aligned images, and 2) achieve exceptional performance on standard benchmarks such as GenEval, DPG- Bench, and WISE, with only 1.1B and 3.1B activated parameters. To support open research and community advancement, we release all model weights, training code, and our curated training datasets (including 23M image-text pairs) at https://github.com/wusize/OpenUni.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 56 of its 65 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

56 of 65 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2025Qwen2.5-VL Technical Reportcited by 5,587
38 more
  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  13. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  16. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  17. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  18. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  19. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  20. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  21. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  22. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  23. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  24. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  25. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  26. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  27. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  28. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
  29. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  30. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  31. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 217
  32. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  33. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  34. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  35. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  36. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  37. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 132
  38. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 29.

46 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  2. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  3. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  4. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  5. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 28
  6. 2026Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generationcited by 20
  7. 2025Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generationcited by 19
  8. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  9. 2025Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generationcited by 17
  10. 2025TwinFlow: Realizing One-step Generation on Large Models with Self-adversarial Flowscited by 17
  11. 2026DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editingcited by 14
  12. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generationcited by 13
34 more
  1. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  2. 2026UniReason 1.0: A Unified Reasoning Framework for World Knowledge Aligned Image Generation and Editingcited by 12
  3. 2025HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generationcited by 11
  4. 2025TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuningcited by 10
  5. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 10
  6. 2025SA-LUT: Spatial Adaptive 4D Look-Up Table for Photorealistic Style Transfercited by 8
  7. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 5
  8. 2025Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editingcited by 4
  9. 2026Scaling GUI Agents with Visual State Transitionscited by 4
  10. 2025Stitch: Training-Free Position Control in Multimodal Diffusion Transformerscited by 3
  11. 2026Duality Models: An Embarrassingly Simple One-step Generation Paradigmcited by 3
  12. 2026MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonalitycited by 3
  13. 2026Steering Visual Generation in Unified Multimodal Models with Understanding Supervisioncited by 3
  14. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  15. 2026HYDRA: Representation Harmonized Tokenization for Multimodal Generation and Understandingcited by 2
  16. 2026Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-trainingcited by 2
  17. 2026TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-trainingcited by 2
  18. 2026Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representationcited by 2
  19. 2026Camera Control for Text-to-Image Generation via Learning Viewpoint Tokenscited by 2
  20. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Modelscited by 2
  21. 2026IV-CoT: Implicit Visual Chain-of-Thought for Structure-Aware Text-to-Image Generationcited by 2
  22. 2025ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interactioncited by 1
  23. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1
  24. 2026LatentUMM: Dual Latent Alignment for Unified Multimodal Modelscited by 1
  25. 2026Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generationcited by 1
  26. 2026SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Modelscited by 1
  27. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  28. 2026InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMscited by 0
  29. 2026Enhancing Alignment for Unified Multimodal Models via Semantically-Grounded Supervisioncited by 0
  30. 2026IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generationcited by 0
  31. 2026Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generationcited by 0
  32. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0
  33. 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
  34. 2026Mutually Adversarial Self-Training with Evolving Data for Unified Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  2. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
  3. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation
  4. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  6. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  7. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
  8. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning