mlsanityData to 14 Oct 2026

OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows

arXiv 2412.01169 ↗First posted 2 Dec 2024cs.MM cs.CV cs.SDcited by 46

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We introduce OmniFlow, a novel generative model designed for any-to-any generation tasks such as text-to-image, text-to-audio, and audio-to-image synthesis. OmniFlow advances the rectified flow (RF) framework used in text-to-image models to handle the joint distribution of multiple modalities. It outperforms previous any-to-any models on a wide range of tasks, such as text-to-image and text-to-audio synthesis. Our work offers three key contributions: First, we extend RF to a multi-modal setting and introduce a novel guidance mechanism, enabling users to flexibly control the alignment between different modalities in the generated outputs. Second, we propose a novel architecture that extends the text-to-image MMDiT architecture of Stable Diffusion 3 and enables audio and text generation. The extended modules can be efficiently pretrained individually and merged with the vanilla text-to-image MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design choices of rectified flow transformers for large-scale audio and text generation, providing valuable insights into optimizing performance across diverse modalities. The Code will be available at https://github.com/jacklishufan/OmniFlows.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 41 of its 52 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

41 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  2. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  3. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models30 Jan

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2022Flow Matching for Generative Modelingcited by 5,702
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
26 more
  1. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  7. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  8. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  9. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  10. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  11. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  13. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  14. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  15. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  16. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  17. 2022AudioGen: Textually Guided Audio Generationcited by 295
  18. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  19. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  20. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  21. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  22. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  23. 2023Equivariant flow matchingcited by 135
  24. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  25. 2024AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Modelscited by 21
  26. 2024SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 21.

46 papers.

Sort:

By area:

  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  2. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 84
  3. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 47
  4. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 44
  5. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 43
  6. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
  7. 2025Continuous Diffusion Model for Language Modelingcited by 31
  8. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 24
  9. 2025Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spacescited by 21
  10. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  11. 2025Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generationcited by 15
  12. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
34 more
  1. 2025Survey on the Evaluation of Generative Models in Musiccited by 10
  2. 2025CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusioncited by 9
  3. 2025RubricRL: Simple Generalizable Rewards for Text-to-Image Generationcited by 7
  4. 2025Flow Straight and Fast in Hilbert Space: Functional Rectified Flowcited by 6
  5. 2025UniF2ace: A Unified Fine-grained Face Understanding and Generation Modelcited by 5
  6. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generationcited by 5
  7. 2026UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generationcited by 5
  8. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  9. 2025UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generationcited by 3
  10. 2026Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detectioncited by 3
  11. 2025Flow Diverse and Efficient: Learning Momentum Flow Matching via Stochastic Velocity Field Samplingcited by 2
  12. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  13. 2026CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusioncited by 2
  14. 2026VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creationcited by 2
  15. 2026MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generationcited by 2
  16. 2025RFMI: Estimating Mutual Information on Rectified Flow for Text-to-Image Alignmentcited by 1
  17. 2025Universal Multi-Domain Translation via Diffusion Routerscited by 1
  18. 2025FlowBind: Efficient Any-to-Any Generation with Bidirectional Flowscited by 1
  19. 2026Any2Any 3D Diffusion Models with Knowledge Transfer: A Radiotherapy Planning Studycited by 1
  20. 2026MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modelingcited by 1
  21. 2025UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generationcited by 0
  22. 2026VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agentscited by 0
  23. 2026Training-free, Perceptually Consistent Low-Resolution Previews with High-Resolution Image for Efficient Workflows of Diffusion Modelscited by 0
  24. 2026AsyncPatch Diffusion: spatially-flexible image generationcited by 0
  25. 2026Joint Flow Matching Enables Continuous Dose-Conditioned Cell Morphingcited by 0
  26. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0
  27. 2026Importance-Aware Low-Rank Distillation of Diffusion Transformerscited by 0
  28. 2026LoGAN: Multilingual Font Localization with Generative Agentscited by 0
  29. 2026BirdsongChat: A Hybrid Multi-Agent Framework for Multimodal Embodied Behavior Simulationcited by 0
  30. 2026Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstructioncited by 0
  31. 2026Off-manifold robustness in synthesizer inversion with joint distribution flow matchingcited by 0
  32. 2026Harnessing Coupled Stream Completion For Human-Object Interaction Modelingcited by 0
  33. 2026Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spacescited by 0
  34. 2026AnchorGen: Anchored Optimization for Customizable Generative 3D Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OmniSonic: Towards Universal and Holistic Audio Generation from Video and Text
  2. 2026OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
  3. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
  4. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching
  5. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
  6. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
  7. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities
  8. 2023Align, Adapt and Inject: Sound-guided Unified Image Generation