mlsanityData to 7 Oct 2026

Scaling Properties of Diffusion Models for Perceptual Tasks

arXiv 2411.08034 ↗First posted 12 Nov 2024cs.CV cs.AIcited by 15

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In this paper, we argue that iterative computation with diffusion models offers a powerful paradigm for not only generation but also visual perception tasks. We unify tasks such as depth estimation, optical flow, and amodal segmentation under the framework of image-to-image translation, and show how diffusion models benefit from scaling training and test-time compute for these perceptual tasks. Through a careful analysis of these scaling properties, we formulate compute-optimal training and inference recipes to scale diffusion models for visual perception tasks. Our models achieve competitive performance to state-of-the-art methods using significantly less data and compute. To access our code and models, see https://scaling-diffusion-perception.github.io .

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 44 of its 61 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

44 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis10 Jul
  2. 2024On the Scalability of Diffusion-based Text-to-Image Generation3 Apr
  3. 2023DDP: Diffusion Model for Dense Visual Prediction30 Mar
  4. 2023Unleashing Text-to-Image Diffusion Models for Visual Perception3 Mar
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2015Variational Inference with Normalizing Flowscited by 2,161
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  12. 2023Consistency Modelscited by 1,778
26 more
  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  4. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  5. 2016Pixel Recurrent Neural Networkscited by 814
  6. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  7. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  8. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  9. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  10. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  11. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  12. 2022Novel View Synthesis with Diffusion Modelscited by 299
  13. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
  14. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  15. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  16. 2021Diffusion Models for Implicit Image Segmentation Ensemblescited by 198
  17. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
  18. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  19. 2023Monocular Depth Estimation using Diffusion Modelscited by 87
  20. 2023DiffusionDepth: Diffusion Denoising Approach for Monocular Depth Estimationcited by 81
  21. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  22. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  23. 2022DiffusionInst: Diffusion Model for Instance Segmentationcited by 49
  24. 2023Semantic Diffusion Network for Semantic Segmentationcited by 30
  25. 2024EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routingcited by 29
  26. 2024SWAG: Storytelling With Action Guidancecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Video models are zero-shot learners and reasonerscited by 208
  2. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  3. 2025The Serial Scaling Hypothesiscited by 11
  4. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  5. 2025LangScene-X: Reconstruct Generalizable 3D Language-Embedded Scenes with TriMap Video Diffusioncited by 6
  6. 2025DvD: Unleashing a Generative Paradigm for Document Dewarping via Coordinates-based Diffusion Modelcited by 4
  7. 2026Motion Attribution for Video Generationcited by 4
  8. 2025Perfect diffusion is mathsfTC0 -- Bad diffusion is Turing-completecited by 3
  9. 2025gen2seg: Generative Models Enable Generalizable Instance Segmentationcited by 2
  10. 2026MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modelingcited by 2
  11. 2025AMix-1: A Pathway to Test-Time Scalable Protein Foundation Modelcited by 1
  12. 2025Training Video Foundation Models with NVIDIA NeMocited by 0
3 more
  1. 2026GenDet: Painting Colored Bounding Boxes on Images via Diffusion Model for Object Detectioncited by 0
  2. 2026MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysiscited by 0
  3. 2026Bridging Video Understanding and Generation in a Unified Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?
  2. 2024Diffusion Models in Low-Level Vision: A Survey
  3. 2024Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models
  4. 2026Abra: Scaling Diffusion Image Training
  5. 2024Towards Precise Scaling Laws for Video Diffusion Transformers
  6. 2024Toward a Diffusion-Based Generalist for Dense Vision Tasks
  7. 2023Harnessing Diffusion Models for Visual Perception with Meta Prompts
  8. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception