mlsanityData to 7 Oct 2026

Optimizing Diffusion Noise Can Serve As Universal Motion Priors

arXiv 2312.11994 ↗First posted 19 Dec 2023cs.CVcited by 97

On the map: Diffusion models → Video generation

Abstract (arXiv)

We propose Diffusion Noise Optimization (DNO), a new method that effectively leverages existing motion diffusion models as motion priors for a wide range of motion-related tasks. Instead of training a task-specific diffusion model for each new task, DNO operates by optimizing the diffusion latent noise of an existing pre-trained text-to-motion model. Given the corresponding latent noise of a human motion, it propagates the gradient from the target criteria defined on the motion space through the whole denoising process to update the diffusion latent noise. As a result, DNO supports any use cases where criteria can be defined as a function of motion. In particular, we show that, for motion editing and control, DNO outperforms existing methods in both achieving the objective and preserving the motion content. DNO accommodates a diverse range of editing modes, including changing trajectory, pose, joint locations, or avoiding newly added obstacles. In addition, DNO is effective in motion denoising and completion, producing smooth and realistic motion from noisy and partial inputs. DNO achieves these results at inference time without the need for model retraining, offering great versatility for any defined reward or loss function on the motion representation.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 78 of its 91 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

78 of 91 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Guided Motion Diffusion for Controllable Human Motion Synthesis21 May
  2. 2023Human Motion Diffusion as a Generative Prior2 Mar
  3. 2022Executing your Commands via Motion Diffusion in Latent Space8 Dec
  4. 2022Pretrained Diffusion Models for Unified Human Motion Synthesis6 Dec
  5. 2022PhysDiff: Physics-Guided Human Motion Diffusion Model5 Dec
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  7. 2022Human Motion Diffusion Model29 Sep
  8. 2022FLAME: Free-form Language-based Motion Synthesis & Editing1 Sep
  9. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug
  10. 2022Video Diffusion Models7 Apr

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
38 more
  1. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  6. 2023Consistency Modelscited by 1,778
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  12. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  13. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  14. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  15. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  16. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  17. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  18. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  19. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  20. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  21. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  22. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  23. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 445
  24. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  25. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  26. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  27. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  28. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  29. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  30. 2022Novel View Synthesis with Diffusion Modelscited by 299
  31. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  32. 2022EDGE: Editable Dance Generation From Musiccited by 284
  33. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  34. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  35. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 243
  36. 2017AttGAN: Facial Attribute Editing by Only Changing What You Wantcited by 241
  37. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  38. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 34; 2026: 40.

97 papers.

Sort:

By area:

  1. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  2. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  3. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  4. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  5. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  6. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 59
  7. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  8. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46
  9. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  10. 2024MaskControl: Spatio-Temporal Control for Masked Motion Synthesiscited by 37
  11. 2025SALAD: Skeleton-aware Latent Diffusion for Text-driven Motion Generation and Editingcited by 37
  12. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
38 more
  1. 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 29
  2. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  3. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  4. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  5. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  6. 2024Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patternscited by 21
  7. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 20
  8. 2024CoMA: Compositional Human Motion Generation with Multi-modal Agentscited by 18
  9. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  10. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 16
  11. 2024COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimationcited by 15
  12. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  13. 2025Nonisotropic Gaussian Diffusion for Realistic 3D Human Motion Predictioncited by 12
  14. 2024RectifID: Personalizing Rectified Flow with Anchored Classifier Guidancecited by 11
  15. 2025Source-Guided Flow Matchingcited by 11
  16. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 10
  17. 2024MoLA: Motion Generation and Editing with Latent Diffusion Enhanced by Adversarial Trainingcited by 9
  18. 2025Motion Synthesis with Sparse and Flexible Keyjoint Controlcited by 9
  19. 2025VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisionscited by 9
  20. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  21. 2025PINO: Person-Interaction Noise Optimization for Long-Duration and Customizable Motion Generation of Arbitrary-Sized Groupscited by 8
  22. 2025Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generationcited by 7
  23. 2025ARFlow: Human Action-Reaction Flow Matching with Physical Guidancecited by 7
  24. 2025Model See Model Do: Speech-Driven Facial Animation with Style Controlcited by 7
  25. 2024Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigationcited by 6
  26. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  27. 2026UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priorscited by 5
  28. 2026ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generationcited by 5
  29. 2025TFG-Flow: Training-free Guidance in Multimodal Generative Flowcited by 4
  30. 2025Generative Motion Infilling From Imprecisely Timed Keyframescited by 4
  31. 2025HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidancecited by 4
  32. 2025PP-Motion: Physical-Perceptual Fidelity Evaluation for Human Motion Generationcited by 4
  33. 2025AHA! Animating Human Avatars in Diverse Scenes with Gaussian Splattingcited by 4
  34. 2025SONIC: Spectral Optimization of Noise for Inpainting with Consistencycited by 4
  35. 2026ProjFlow: Projection Sampling with Flow Matching for Zero-Shot Exact Spatial Motion Controlcited by 4
  36. 2024GPT-Connect: Interaction between Text-Driven Human Motion Generator and 3D Scenes in a Training-free Mannercited by 3
  37. 2025Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Modelscited by 3
  38. 2024Recovering Complete Actions for Cross-dataset Skeleton Action Recognitioncited by 2

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation
  2. 2022Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Models
  3. 2023TEDi: Temporally-Entangled Diffusion for Long-Term Motion Synthesis
  4. 2026Parasitic Co-Denoising: Unlocking 3D Human Motion Generation in a Frozen Video Diffusion Model
  5. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning
  6. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models
  7. 2026Denoising as Path Planning: Training-Free Acceleration of Diffusion Models with DPCache
  8. 2026From Diffusion to Flow: Efficient Motion Generation in MotionGPT3