# Scaling Properties of Diffusion Models for Perceptual Tasks

arXiv: https://arxiv.org/abs/2411.08034 · HTML: https://mlsanity.com/paper/2411.08034/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 44 of its 61 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 44)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints](https://mlsanity.com/paper/2212.05055/) (2022)

## Built on by (most cited first, 15 of 15)

- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning](https://mlsanity.com/paper/2511.19515/) (2025)
- [The Serial Scaling Hypothesis](https://mlsanity.com/paper/2507.12549/) (2025)
- [Can Video Diffusion Model Reconstruct 4D Geometry?](https://mlsanity.com/paper/2503.21082/) (2025)
- [LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusion](https://mlsanity.com/paper/2507.02813/) (2025)
- [Motion Attribution for Video Generation](https://mlsanity.com/paper/2601.08828/) (2026)
- [DvD: Unleashing a Generative Paradigm for Document Dewarping via Coordinates-based Diffusion Model](https://mlsanity.com/paper/2505.21975/) (2025)
- [Perfect diffusion is mathsfTC0 -- Bad diffusion is Turing-complete](https://mlsanity.com/paper/2507.12469/) (2025)
- [MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling](https://mlsanity.com/paper/2604.02817/) (2026)
- [gen2seg: Generative Models Enable Generalizable Instance Segmentation](https://mlsanity.com/paper/2505.15263/) (2025)
- [AMix-1: A Pathway to Test-Time Scalable Protein Foundation Model](https://mlsanity.com/paper/2507.08920/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)
- [Training Video Foundation Models with NVIDIA NeMo](https://mlsanity.com/paper/2503.12964/) (2025)
- [GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detection](https://mlsanity.com/paper/2601.07273/) (2026)
- [Bridging Video Understanding and Generation in a Unified Framework](https://mlsanity.com/paper/2606.31326/) (2026)

## Abstract (arXiv)

In this paper, we argue that iterative computation with diffusion models offers a powerful paradigm for not only generation but also visual perception tasks. We unify tasks such as depth estimation, optical flow, and amodal segmentation under the framework of image-to-image translation, and show how diffusion models benefit from scaling training and test-time compute for these perceptual tasks. Through a careful analysis of these scaling properties, we formulate compute-optimal training and inference recipes to scale diffusion models for visual perception tasks. Our models achieve competitive performance to state-of-the-art methods using significantly less data and compute. To access our code and models, see https://scaling-diffusion-perception.github.io .