# Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution

arXiv: https://arxiv.org/abs/2412.15213 · HTML: https://mlsanity.com/paper/2412.15213/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 83 of its 98 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 83)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)

## Built on by (most cited first, 26 of 26)

- [Stochastic Process Learning via Operator Flow Matching](https://mlsanity.com/paper/2501.04126/) (2025)
- [Better Source, Better Flow: Learning Condition-Dependent Source Distribution for Flow Matching](https://mlsanity.com/paper/2602.05951/) (2026)
- [Autoregressive Image Generation with Masked Bit Modeling](https://mlsanity.com/paper/2602.09024/) (2026)
- [BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving](https://mlsanity.com/paper/2604.10856/) (2026)
- [Exploring Cross-Modal Flows for Few-Shot Learning](https://mlsanity.com/paper/2510.14543/) (2025)
- [SCFlow: Implicitly Learning Style and Content Disentanglement with Flow Models](https://mlsanity.com/paper/2508.03402/) (2025)
- [Diffusion Bridge or Flow Matching? A Unifying Framework and Comparative Analysis](https://mlsanity.com/paper/2509.24531/) (2025)
- [Learning by Neighbor-Aware Semantics, Deciding by Open-form Flows: Towards Robust Zero-Shot Skeleton Action Recognition](https://mlsanity.com/paper/2511.09388/) (2025)
- [Environment-Aware Channel Inference via Cross-Modal Flow: From Multimodal Sensing to Wireless Channels](https://mlsanity.com/paper/2512.04966/) (2025)
- [FlowComposer: Composable Flows for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2603.16641/) (2026)
- [Spatiotemporal Imputation with Graph-Informed Flow Matching](https://mlsanity.com/paper/2606.06682/) (2026)
- [FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching](https://mlsanity.com/paper/2604.06757/) (2026)
- [FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2607.02284/) (2026)
- [TriDiff-4D: Fast 4D Generation through Diffusion-based Triplane Re-posing](https://mlsanity.com/paper/2511.16662/) (2025)
- [FlowBind: Efficient Any-to-Any Generation with Bidirectional Flows](https://mlsanity.com/paper/2512.15420/) (2025)
- [Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification](https://mlsanity.com/paper/2512.16921/) (2025)
- [Improving Diffusion Generalization with Weak-to-Strong Segmented Guidance](https://mlsanity.com/paper/2603.20584/) (2026)
- [Bridging Vision and Language Concepts through Optimal Transport Semantic Flow](https://mlsanity.com/paper/2606.26891/) (2026)
- [Generative Point Tracking with Flow Matching](https://mlsanity.com/paper/2510.20951/) (2025)
- [CrossFlowDG: Bridging the Modality Gap with Cross-modal Flow Matching for Domain Generalization](https://mlsanity.com/paper/2604.16892/) (2026)
- [GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention](https://mlsanity.com/paper/2606.06249/) (2026)
- [Generative Lane Topology Reasoning via Autoregressive Model with Geometry Prior](https://mlsanity.com/paper/2606.31814/) (2026)
- [Contrastive-Augmented Flow Matching for Style-Content Disentanglement](https://mlsanity.com/paper/2607.12404/) (2026)
- [STEREOFLOW: Progressive Stereo Matching with StereoDiT and Transition Flow Matching](https://mlsanity.com/paper/2607.19986/) (2026)
- [FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation](https://mlsanity.com/paper/2609.16591/) (2026)
- [Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces](https://mlsanity.com/paper/2609.40362/) (2026)

## Abstract (arXiv)

Diffusion models, and their generalization, flow matching, have had a remarkable impact on the field of media generation. Here, the conventional approach is to learn the complex mapping from a simple source distribution of Gaussian noise to the target media distribution. For cross-modal tasks such as text-to-image generation, this same mapping from noise to image is learnt whilst including a conditioning mechanism in the model. One key and thus far relatively unexplored feature of flow matching is that, unlike Diffusion models, they are not constrained for the source distribution to be noise. Hence, in this paper, we propose a paradigm shift, and ask the question of whether we can instead train flow matching models to learn a direct mapping from the distribution of one modality to the distribution of another, thus obviating the need for both the noise distribution and conditioning mechanism. We present a general and simple framework, CrossFlow, for cross-modal flow matching. We show the importance of applying Variational Encoders to the input data, and introduce a method to enable Classifier-free guidance. Surprisingly, for text-to-image, CrossFlow with a vanilla transformer without cross attention slightly outperforms standard flow matching, and we show that it scales better with training steps and model size, while also allowing for interesting latent arithmetic which results in semantically meaningful edits in the output space. To demonstrate the generalizability of our approach, we also show that CrossFlow is on par with or outperforms the state-of-the-art for various cross-modal / intra-modal mapping tasks, viz. image captioning, depth estimation, and image super-resolution. We hope this paper contributes to accelerating progress in cross-modal media generation.