mlsanityData to 7 Oct 2026

Learning Semantic Latent Directions for Accurate and Controllable Human Motion Prediction

arXiv 2407.11494 ↗First posted 16 Jul 2024cs.CVcited by 14

On the map: Diffusion models → Video generation

Abstract (arXiv)

In the realm of stochastic human motion prediction (SHMP), researchers have often turned to generative models like GANS, VAEs and diffusion models. However, most previous approaches have struggled to accurately predict motions that are both realistic and coherent with past motion due to a lack of guidance on the latent distribution. In this paper, we introduce Semantic Latent Directions (SLD) as a solution to this challenge, aiming to constrain the latent space to learn meaningful motion semantics and enhance the accuracy of SHMP. SLD defines a series of orthogonal latent directions and represents the hypothesis of future motion as a linear combination of these directions. By creating such an information bottleneck, SLD excels in capturing meaningful motion semantics, thereby improving the precision of motion predictions. Moreover, SLD offers controllable prediction capabilities by adjusting the coefficients of the latent directions during the inference phase. Expanding on SLD, we introduce a set of motion queries to enhance the diversity of predictions. By aligning these motion queries with the SLD space, SLD is further promoted to more accurate and coherent motion predictions. Through extensive experiments conducted on widely used benchmarks, we showcase the superiority of our method in accurately predicting motions while maintaining a balance of realism and diversity. Our code and pretrained models are available at https://github.com/GuoweiXu368/SLD-HMP.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 40 of its 59 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

40 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction25 Nov

Other references · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  7. 2017Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecastingcited by 1,125
  8. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  9. 2017On human motion prediction using recurrent neural networkscited by 336
  10. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  11. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 251
  12. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
27 more
  1. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  2. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  3. 2021Character Controllers Using Motion VAEscited by 182
  4. 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
  5. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  6. 2021MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Predictioncited by 100
  7. 2018MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamicscited by 86
  8. 2021Space-Time-Separable Graph Convolutional Network for Pose Forecastingcited by 84
  9. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  10. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  11. 2017DeLiGAN : Generative Adversarial Networks for Diverse and Limited Datacited by 77
  12. 2021Generating Smooth Pose Sequences for Diverse Human Motion Predictioncited by 66
  13. 2018Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objectivecited by 56
  14. 2022Spatio-Temporal Gating-Adjacency GCN for Human Motion Predictioncited by 48
  15. 2021Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformerscited by 46
  16. 2021Multi-level Motion Attention for Human Motion Predictioncited by 35
  17. 2023Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchorscited by 34
  18. 2022Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Spacecited by 32
  19. 2023DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Modelscited by 30
  20. 2022Human Joint Kinematics Diffusion-Refinement for Stochastic Motion Predictioncited by 28
  21. 2024Diffusion Model with Cross Attention as an Inductive Bias for Disentanglementcited by 27
  22. 2021InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generationcited by 18
  23. 2018Structure from Recurrent Motion: From Rigidity to Recurrencycited by 16
  24. 2023Self-Supervised Video Representation Learning via Latent Time Navigationcited by 11
  25. 2022Learning Disentangled Representations for Controllable Human Motion Predictioncited by 5
  26. 2021DMS-GCN: Dynamic Mutiscale Spatiotemporal Graph Convolutional Networks for Human Motion Predictioncited by 4
  27. 2023Multiscale Residual Learning of Graph Convolutional Sequence Chunks for Human Motion Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Nonisotropic Gaussian Diffusion for Realistic 3D Human Motion Predictioncited by 12
  2. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  3. 2024MotionWavelet: Human Motion Prediction via Wavelet Manifold Learningcited by 8
  4. 2024MotionMap: Representing Multimodality in Human Pose Forecastingcited by 7
  5. 2025CacheFlow: Fast Human Motion Prediction by Cached Normalizing Flowcited by 4
  6. 2025OmniMotion-X: Versatile Multimodal Whole-Body Motion Generationcited by 2
  7. 2025Controllable Generative Trajectory Prediction via Weak Preference Alignmentcited by 0
  8. 2026SimpliHuMoN: Simplifying Human Motion Predictioncited by 0
  9. 2026KHMP: Frequency-Domain Kalman Refinement for High-Fidelity Human Motion Predictioncited by 0
  10. 2026Capturing Uncertainty in Human Motion for Representation Learning in Soccercited by 0
  11. 2026SPARC: Single-Pass Scaling for Motion Forecasting with Conformal Bayesian Last Layerscited by 0
  12. 2026Gaussian-Mixture Latent Flow for Stochastic 3D Human Motion Predictioncited by 0
2 more
  1. 2026Multi-Person Human Motion Forecasting in Complex Scenescited by 0
  2. 2026From Where to How: Continuous 4D Interaction Forecasting from Egocentric Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction
  2. 2026Semantic Belief-State World Model for 3D Human Motion Prediction
  3. 2023Stochastic Multi-Person 3D Motion Forecasting
  4. 2022Executing your Commands via Motion Diffusion in Latent Space
  5. 2025HumanCM: One Step Human Motion Prediction
  6. 2024LEAD: Latent Realignment for Human Motion Diffusion
  7. 2024Length-Aware Motion Synthesis via Latent Diffusion
  8. 2024LS-GAN: Human Motion Synthesis with Latent-space GANs