mlsanityData to 7 Oct 2026

Behavior-Driven Synthesis of Human Dynamics

arXiv 2103.04677 ↗First posted 8 Mar 2021cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

Generating and representing human behavior are of major importance for various computer vision applications. Commonly, human video synthesis represents behavior as sequences of postures while directly predicting their likely progressions or merely changing the appearance of the depicted persons, thus not being able to exercise control over their actual behavior during the synthesis process. In contrast, controlled behavior synthesis and transfer across individuals requires a deep understanding of body dynamics and calls for a representation of behavior that is independent of appearance and also of specific postures. In this work, we present a model for human behavior synthesis which learns a dedicated representation of human dynamics independent of postures. Using this representation, we are able to change the behavior of a person depicted in an arbitrary posture, or to even directly transfer behavior observed in a given video sequence. To this end, we propose a conditional variational framework which explicitly disentangles posture from behavior. We demonstrate the effectiveness of our approach on this novel task, evaluating capturing, transferring, and sampling fine-grained, diverse behavior, both quantitatively and qualitatively. Project page is available at https://cutt.ly/5l7rXEp

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 53 of its 73 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

53 of 73 references.

References · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  10. 2016Density estimation using Real NVPcited by 2,029
  11. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  12. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
38 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  2. 2018MINE: Mutual Information Neural Estimationcited by 680
  3. 2015Generating Sentences from a Continuous Spacecited by 607
  4. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  5. 2018Real-world Anomaly Detection in Surveillance Videoscited by 570
  6. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  7. 2018Video-to-Video Synthesiscited by 388
  8. 2016Generating Videos with Scene Dynamicscited by 376
  9. 2017On human motion prediction using recurrent neural networkscited by 336
  10. 2018Everybody Dance Nowcited by 313
  11. 2017Pose Guided Person Image Generationcited by 251
  12. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  13. 2019Few-shot Video-to-Video Synthesiscited by 196
  14. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  15. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  16. 2016Variational Lossy Autoencodercited by 175
  17. 2019Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesiscited by 161
  18. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  19. 2017InfoVAE: Information Maximizing Variational Autoencoderscited by 124
  20. 2018A Variational U-Net for Conditional Appearance and Shape Generationcited by 124
  21. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  22. 2017Compressed Video Action Recognitioncited by 102
  23. 2017Disentangled Person Image Generationcited by 98
  24. 2018Recycle-GAN: Unsupervised Video Retargetingcited by 89
  25. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  26. 2016Disentangling factors of variation in deep representations using adversarial trainingcited by 87
  27. 2018MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamicscited by 86
  28. 2016Deep Feature Interpolation for Image Content Changescited by 80
  29. 2018Pose Flow: Efficient Online Pose Trackingcited by 80
  30. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  31. 2015Learning Deep Representations of Appearance and Motion for Anomalous Event Detectioncited by 75
  32. 2018Pose Guided Human Video Generationcited by 67
  33. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  34. 2018Action-Agnostic Human Pose Forecastingcited by 56
  35. 2018Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoderscited by 49
  36. 2019Learning Representations by Maximizing Mutual Information in Variational Autoencoderscited by 20
  37. 2019Vid2Game: Controllable Characters Extracted from Real-World Videoscited by 18
  38. 2019Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesiscited by 12

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 4; 2023: 3; 2024: 1; 2025: 1.

14 papers.

Sort:

By area:

  1. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  2. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Predictioncited by 82
  3. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  4. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  5. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  6. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  7. 2023Interactive Humanoid: Online Full-Body Motion Reaction Synthesis with Social Affordance Canonicalization and Forecastingcited by 27
  8. 2023CoMusion: Towards Consistent Stochastic Human Motion Prediction via Motion Diffusioncited by 21
  9. 2021Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Gamescited by 17
  10. 2022SCVRL: Shuffled Contrastive Video Representation Learningcited by 14
  11. 2024MotionWavelet: Human Motion Prediction via Wavelet Manifold Learningcited by 8
  12. 2022Learning Disentangled Representations for Controllable Human Motion Predictioncited by 5
2 more
  1. 2022FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observationscited by 5
  2. 2025Stochastic Human Motion Prediction with Memory of Action Transition and Action Characteristiccited by 0