mlsanityData to 7 Oct 2026

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

arXiv 2304.01893 ↗First posted 4 Apr 2023cs.CV cs.GR cs.LGcited by 137

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. We draw on recent advances in guided diffusion modeling to achieve test-time controllability of trajectories, which is normally only associated with rule-based systems. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups while accounting for the surrounding environment context. This trajectory diffusion model is integrated with a novel physics-based humanoid controller to form a closed-loop, full-body pedestrian animation system capable of placing large crowds in a simulated environment with varying terrains. We further propose utilizing the value function learned during RL training of the animation controller to guide diffusion to produce trajectories better suited for particular scenarios such as collision avoidance and traversing uneven terrain. Video results are available on the project page at https://nv-tlabs.github.io/trace-pace .

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 44 of its 76 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

44 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Guided Conditional Diffusion for Controllable Traffic Simulation31 Oct

Other references · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
31 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  6. 2022On Distillation of Guided Diffusion Modelscited by 721
  7. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  8. 2021Score-based Generative Modeling in Latent Spacecited by 634
  9. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
  10. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  11. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  12. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  13. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  14. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  15. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  16. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  17. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  18. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  19. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 184
  20. 2021Character Controllers Using Motion VAEscited by 182
  21. 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 166
  22. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  23. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  24. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  25. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  26. 2022Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotioncited by 98
  27. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  28. 2021The Wanderings of Odysseus in 3D Scenescited by 59
  29. 2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57
  30. 2018Learning Symmetric and Low-energy Locomotioncited by 53
  31. 2022Embodied Scene-aware Human Pose Estimationcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 28; 2024: 40; 2025: 48; 2026: 21.

137 papers.

Sort:

By area:

  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  2. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  3. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  4. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  5. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  6. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  7. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  8. 2023Controllable Human-Object Interaction Synthesiscited by 107
  9. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 105
  10. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  11. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  12. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
38 more
  1. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  2. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  3. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  4. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  5. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  6. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  7. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 47
  8. 2024RoHM: Robust Human Motion Reconstruction via Diffusioncited by 46
  9. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  10. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  11. 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 43
  12. 2023Iterative Motion Editing with Natural Languagecited by 39
  13. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  14. 2024MaskControl: Spatio-Temporal Control for Masked Motion Synthesiscited by 37
  15. 2023Boosting Continuous Control with Consistency Policycited by 36
  16. 2023SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversariescited by 36
  17. 2024SingularTrajectory: Universal Trajectory Predictor Using Diffusion Modelcited by 35
  18. 2026Kimodo: Scaling Controllable Human Motion Generationcited by 33
  19. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  20. 2024SMPLOlympics: Sports Environments for Physically Simulated Humanoidscited by 29
  21. 2023SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Predictioncited by 27
  22. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  23. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  24. 2025SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Predictioncited by 26
  25. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  26. 2023trajdata: A Unified Interface to Multiple Human Trajectory Datasetscited by 25
  27. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  28. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  29. 2023InsActor: Instruction-driven Physics-based Characterscited by 23
  30. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  31. 2023DICE: Diverse Diffusion Model with Scoring for Trajectory Predictioncited by 18
  32. 2024Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generationcited by 18
  33. 2023A Diffusion-Model of Joint Interactive Navigationcited by 16
  34. 2025Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modelingcited by 15
  35. 2024Diffusion-Based Environment-Aware Trajectory Predictioncited by 14
  36. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  37. 2023HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Modelscited by 13
  38. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PDP: Physics-Based Character Animation via Diffusion Policy
  2. 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control
  3. 2026PhysMoDPO: Physically-Plausible Humanoid Motion with Preference Optimization
  4. 2026DreamControl-v2: Simpler and Scalable Autonomous Humanoid Skills via Trainable Guided Diffusion Priors
  5. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusion
  6. 2025Physically Consistent Humanoid Loco-Manipulation using Latent Diffusion Models
  7. 2026PASSAGE: Scaling Scene-Aligned Motion Learning for Perceptive Humanoid Traversal in Cluttered Environments ↗
  8. 2024Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models