# One Diffusion to Generate Them All

arXiv: https://arxiv.org/abs/2411.16318 · HTML: https://mlsanity.com/paper/2411.16318/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 63 of its 76 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 63)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)

## Built on by (most cited first, 30 of 41)

- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer](https://mlsanity.com/paper/2503.07027/) (2025)
- [Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding](https://mlsanity.com/paper/2510.06308/) (2025)
- [Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling](https://mlsanity.com/paper/2507.17801/) (2025)
- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [OmniVDiff: Omni Controllable Video Diffusion for Generation and Understanding](https://mlsanity.com/paper/2504.10825/) (2025)
- [UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing](https://mlsanity.com/paper/2503.12652/) (2025)
- [Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering](https://mlsanity.com/paper/2508.14461/) (2025)
- [Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision](https://mlsanity.com/paper/2504.04903/) (2025)
- [Motion-Aware Generative Frame Interpolation](https://mlsanity.com/paper/2501.03699/) (2025)
- [OmniTry: Virtual Try-On Anything without Masks](https://mlsanity.com/paper/2508.13632/) (2025)
- [MUSAR: Exploring Multi-Subject Customization from Single-Subject Dataset via Attention Routing](https://mlsanity.com/paper/2505.02823/) (2025)
- [MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing](https://mlsanity.com/paper/2502.21291/) (2025)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans](https://mlsanity.com/paper/2506.20879/) (2025)
- [Jodi: Unification of Visual Generation and Understanding via Joint Modeling](https://mlsanity.com/paper/2505.19084/) (2025)
- [The Consistency Critic: Correcting Inconsistencies in Generated Images via Reference-Guided Attentive Alignment](https://mlsanity.com/paper/2511.20614/) (2025)
- [MultiCrafter: High-Fidelity Multi-Subject Generation via Disentangled Attention and Identity-Aware Preference Alignment](https://mlsanity.com/paper/2509.21953/) (2025)
- [Trade-offs in Image Generation: How Do Different Dimensions Interact?](https://mlsanity.com/paper/2507.22100/) (2025)
- [LaVieID: Local Autoregressive Diffusion Transformers for Identity-Preserving Video Creation](https://mlsanity.com/paper/2508.07603/) (2025)
- [DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2603.08090/) (2026)
- [UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors](https://mlsanity.com/paper/2605.00658/) (2026)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets](https://mlsanity.com/paper/2506.08013/) (2025)
- [More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models](https://mlsanity.com/paper/2510.23574/) (2025)
- [ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Control](https://mlsanity.com/paper/2603.14209/) (2026)
- [Camera Control for Text-to-Image Generation via Learning Viewpoint Tokens](https://mlsanity.com/paper/2604.19954/) (2026)
- [ShaLa: Multimodal Shared Latent Space Modelling](https://mlsanity.com/paper/2508.17376/) (2025)
- [DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models](https://mlsanity.com/paper/2509.22793/) (2025)
- [Universal Multi-Domain Translation via Diffusion Routers](https://mlsanity.com/paper/2510.03252/) (2025)

## Abstract (arXiv)

We introduce OneDiffusion, a versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps, while also handling tasks like image deblurring, upscaling, and reverse processes such as depth estimation and segmentation. Additionally, OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs. Our model takes a straightforward yet effective approach by treating all tasks as frame sequences with varying noise scales during training, allowing any frame to act as a conditioning image at inference time. Our unified training framework removes the need for specialized architectures, supports scalable multi-task training, and adapts smoothly to any resolution, enhancing both generalization and scalability. Experimental results demonstrate competitive performance across tasks in both generation and prediction such as text-to-image, multiview generation, ID preservation, depth estimation and camera pose estimation despite relatively small training dataset. Our code and checkpoint are freely available at https://github.com/lehduong/OneDiffusion