mlsanityData to 7 Oct 2026

NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis

arXiv 2307.07511 ↗First posted 14 Jul 2023cs.CVcited by 78

On the map: Diffusion models → Video generation

Abstract (arXiv)

We address the problem of generating realistic 3D motions of humans interacting with objects in a scene. Our key idea is to create a neural interaction field attached to a specific object, which outputs the distance to the valid interaction manifold given a human pose as input. This interaction field guides the sampling of an object-conditioned human motion diffusion model, so as to encourage plausible contacts and affordance semantics. To support interactions with scarcely available data, we propose an automated synthetic data pipeline. For this, we seed a pre-trained motion model, which has priors for the basics of human movement, with interaction-specific anchor poses extracted from limited motion capture data. Using our guided diffusion model trained on generated synthetic data, we synthesize realistic motions for sitting and lifting with several objects, outperforming alternative approaches in terms of motion quality and successful action completion. We call our framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 52 of its 69 references that are in the map. First posted on arXiv 14 Jul 2023.

Builds on

52 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Human Motion Diffusion as a Generative Prior2 Mar
  2. 2022Human Motion Diffusion Model29 Sep
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug
  4. 2022Video Diffusion Models7 Apr

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  9. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2021Neural Fields in Visual Computing and Beyondcited by 539
  12. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
36 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  3. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  4. 2022EDGE: Editable Dance Generation From Musiccited by 284
  5. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  6. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  7. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  8. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  9. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  10. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  11. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  12. 2021DC3: A learning method for optimization with hard constraintscited by 181
  13. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  14. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  15. 2022HUMANISE: Language-conditioned Human Motion Generation in 3D Scenescited by 156
  16. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  17. 2021Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensorscited by 137
  18. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  19. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  20. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  21. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  22. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  23. 2023Synthesizing Physical Character-Scene Interactionscited by 102
  24. 2022Towards Diverse and Natural Scene-aware 3D Human Motion Synthesiscited by 91
  25. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  26. 2019Context-aware Human Motion Predictioncited by 83
  27. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  28. 2022Full-Body Articulated Human-Object Interactioncited by 76
  29. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  30. 2021The Wanderings of Odysseus in 3D Scenescited by 59
  31. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  32. 2019Learning to Sit: Synthesizing Human-Chair Interactions via Hierarchical Controlcited by 39
  33. 2023Generating Continual Human Motion in Diverse 3D Scenescited by 39
  34. 2022Neural Grasp Distance Fields for Robot Manipulationcited by 31
  35. 2023Scene Synthesis from Human Motioncited by 20
  36. 2022POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networkscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 28; 2025: 27; 2026: 11.

78 papers.

Sort:

By area:

  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  2. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  3. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  4. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  5. 2023Controllable Human-Object Interaction Synthesiscited by 107
  6. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  7. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  8. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  9. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  10. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  11. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  12. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
38 more
  1. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  2. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  3. 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
  4. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  5. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  6. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  7. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  8. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  9. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  10. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  11. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  12. 2024SMooDi: Stylized Motion Diffusion Modelcited by 37
  13. 2024MaskControl: Spatio-Temporal Control for Masked Motion Synthesiscited by 37
  14. 2024HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objectscited by 34
  15. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  16. 2024InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusioncited by 28
  17. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  18. 2024ZeroHSI: Zero-Shot 4D Human-Scene Interaction by Video Generationcited by 25
  19. 2025A Survey on Human Interaction Motion Generationcited by 25
  20. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  21. 2023InterControl: Zero-shot Human Interaction Generation by Controlling Every Jointcited by 22
  22. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  23. 2023OmniMotionGPT: Animal Motion Generation with Limited Datacited by 18
  24. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  25. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 17
  26. 2025UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generationcited by 17
  27. 2025SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidancecited by 16
  28. 2024A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signalscited by 15
  29. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  30. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
  31. 2024Diffgrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Modelcited by 13
  32. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  33. 2025SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarioscited by 11
  34. 2024Physics-based Scene Layout Generation from Human Motioncited by 10
  35. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 10
  36. 2025Motion Synthesis with Sparse and Flexible Keyjoint Controlcited by 9
  37. 2025Motion Generation: A Survey of Generative Approaches and Benchmarkscited by 8
  38. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022NIFT: Neural Interaction Field and Template for Object Manipulation
  2. 2023Object Motion Guided Human Motion Synthesis
  3. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objects
  4. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Models
  5. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization
  6. 2024Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models
  7. 2023Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models
  8. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance