mlsanityData to 14 Oct 2026

Transfer between Modalities with MetaQueries

arXiv 2504.06256 ↗First posted 8 Apr 2025cs.CVcited by 236

Not on the map. Why?

Cited by 45 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Unified multimodal models aim to integrate understanding (text output) and generation (pixel output), but aligning these different modalities within a single architecture often demands complex training recipes and careful data balancing. We introduce MetaQueries, a set of learnable queries that act as an efficient interface between autoregressive multimodal LLMs (MLLMs) and diffusion models. MetaQueries connects the MLLM's latents to the diffusion decoder, enabling knowledge-augmented image generation by leveraging the MLLM's deep understanding and reasoning capabilities. Our method simplifies training, requiring only paired image-caption data and standard diffusion objectives. Notably, this transfer is effective even when the MLLM backbone remains frozen, thereby preserving its state-of-the-art multimodal understanding capabilities while achieving strong generative performance. Additionally, our method is flexible and can be easily instruction-tuned for advanced applications such as image editing and subject-driven generation.

Key facts. Cited by 236 papers in the map (data through 14 Oct 2026). Builds on 55 of its 63 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

55 of 63 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  12. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  13. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  14. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  15. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  16. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  17. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  18. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  19. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
  22. 2024OmniGen: Unified Image Generationcited by 394
  23. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  24. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  25. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  26. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  27. 2023Generating Images with Multimodal Language Modelscited by 260
  28. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  29. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  30. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  31. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 206
  32. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  33. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  34. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  35. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  36. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 132
  37. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  38. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 111; 2026: 125.

236 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  2. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  3. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  4. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  5. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  6. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
  7. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  8. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
  9. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  10. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
  11. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  12. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 100
38 more
  1. 2025LongCat-Image Technical Reportcited by 100
  2. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  3. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
  4. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  5. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 82
  6. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 72
  7. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 66
  8. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  9. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 57
  10. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  11. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 56
  12. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 56
  13. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  14. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 54
  15. 2025SeedEdit 3.0: Fast and High-Quality Generative Image Editingcited by 53
  16. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 53
  17. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 52
  18. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  19. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  20. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 49
  21. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  22. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  23. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  24. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 41
  25. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  26. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  27. 2026Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidancecited by 36
  28. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  29. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 32
  30. 2025Does FLUX Already Know How to Perform Physically Plausible Image Composition?cited by 32
  31. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 32
  32. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 31
  33. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  34. 2025Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizercited by 28
  35. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 28
  36. 2026RLDX-1 Technical Reportcited by 28
  37. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 27
  38. 2026VINO: A Unified Visual Generator with Interleaved OmniModal Contextcited by 26

50 most cited of 236 All 236 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Diffusion for Unified Image Generation and Understanding
  2. 2025Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
  3. 2025Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion
  4. 2025MammothModa2: A Unified AR-Diffusion Framework for Multimodal Understanding and Generation
  5. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
  6. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
  7. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs
  8. 2025I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models