mlsanityData to 7 Oct 2026

Data-Driven Stochastic Motion Evaluation and Optimization with Image by Spatially-Aligned Temporal Encoding

arXiv 2302.05041 ↗First posted 10 Feb 2023cs.CV cs.LG cs.ROcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper proposes a probabilistic motion prediction method for long motions. The motion is predicted so that it accomplishes a task from the initial state observed in the given image. While our method evaluates the task achievability by the Energy-Based Model (EBM), previous EBMs are not designed for evaluating the consistency between different domains (i.e., image and motion in our method). Our method seamlessly integrates the image and motion data into the image feature domain by spatially-aligned temporal encoding so that features are extracted along the motion trajectory projected onto the image. Furthermore, this paper also proposes a data-driven motion optimization method, Deep Motion Optimizer (DMO), that works with EBM for motion prediction. Different from previous gradient-based optimizers, our self-supervised DMO alleviates the difficulty of hyper-parameter tuning to avoid local minima. The effectiveness of the proposed method is demonstrated with a variety of experiments with similar SOTA methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 45 of its 52 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

45 of 52 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2017Improved Training of Wasserstein GANscited by 2,223
  10. 2016Density estimation using Real NVPcited by 2,029
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
33 more
  1. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  2. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  3. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  5. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  6. 2013Network In Networkcited by 600
  7. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  8. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  9. 2021Implicit Behavioral Cloningcited by 442
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2017On human motion prediction using recurrent neural networkscited by 336
  12. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  13. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  14. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  15. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  16. 2020Transformer Networks for Trajectory Forecastingcited by 180
  17. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  18. 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
  19. 2018Human Motion Prediction via Spatio-Temporal Inpaintingcited by 114
  20. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  21. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  22. 2017Robust Imitation of Diverse Behaviorscited by 85
  23. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  24. 2021Generating Smooth Pose Sequences for Diverse Human Motion Predictioncited by 66
  25. 2021Trajectory Prediction with Latent Belief Energy-Based Modelcited by 60
  26. 2019Imitation Learning for Human Pose Predictioncited by 44
  27. 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
  28. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43
  29. 2021Transformer-based deep imitation learning for dual-arm robot manipulationcited by 31
  30. 2017A Data-driven Model for Interaction-aware Pedestrian Motion Prediction in Object Cluttered Environmentscited by 23
  31. 2018Adversarial Contrastive Estimationcited by 23
  32. 2021FloMo: Tractable Motion Prediction with Normalizing Flowscited by 21
  33. 2022MotionAug: Augmentation with Physical Correction for Human Motion Predictioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 0; 2025: 0; 2026: 1.

3 papers.

  1. 2023R2-Diff: Denoising by diffusion as a refinement of retrieved motion for image-based motion predictioncited by 3
  2. 2023Multimodal Active Measurement for Human Mesh Recovery in Close Proximitycited by 2
  3. 2026Energy-Regularized Imitation Learning for Force- and Work-Aware Robotic Manipulationcited by 0