mlsanityData to 7 Oct 2026

Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation

arXiv 2110.11680 ↗First posted 22 Oct 2021cs.CV cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Several video-based 3D pose and shape estimation algorithms have been proposed to resolve the temporal inconsistency of single-image-based methods. However it still remains challenging to have stable and accurate reconstruction. In this paper, we propose a new framework Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation (DTS-VIBE), to generate 3D human pose and mesh from RGB videos. We reformulate the task as a multi-modality problem that fuses RGB and optical flow for more reliable estimation. In order to fully utilize both sensory modalities (RGB or optical flow), we train a two-stream temporal network based on transformer to predict SMPL parameters. The supplementary modality, optical flow, helps to maintain temporal consistency by leveraging motion knowledge between two consecutive frames. The proposed algorithm is extensively evaluated on the Human3.6 and 3DPW datasets. The experimental results show that it outperforms other state-of-the-art methods by a significant margin.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  6. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  7. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  8. 2017End-to-end Recovery of Human Shape and Posecited by 883
  9. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  10. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  11. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  12. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
20 more
  1. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  2. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  3. 2018Learning 3D Human Dynamics from Videocited by 315
  4. 2020I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Imagecited by 274
  5. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  6. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  7. 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
  8. 2020STAR: Sparse Trained Articulated Human Body Regressorcited by 194
  9. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  10. 2018BodyNet: Volumetric Inference of 3D Human Body Shapescited by 141
  11. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  12. 2019Exploiting temporal context for 3D human pose estimation in the wildcited by 110
  13. 20203D Human Motion Estimation via Motion Compression and Refinementcited by 98
  14. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  15. 2017Exploiting temporal information for 3D pose estimationcited by 89
  16. 2019Sim2real transfer learning for 3D human pose estimation: motion to the rescuecited by 75
  17. 2018Supervision-by-Registration: An Unsupervised Approach to Improve the Precision of Facial Landmark Detectorscited by 49
  18. 2016Deep Temporal Linear Encoding Networkscited by 27
  19. 2019DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Imagecited by 20
  20. 2017Deep Local Video Feature for Action Recognitioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 5; 2025: 0; 2026: 1.

7 papers.

  1. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  2. 2024PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videoscited by 27
  3. 2024Towards Open Domain Text-Driven Synthesis of Multi-Person Motionscited by 27
  4. 2024STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusioncited by 9
  5. 2024Multi-RoI Human Mesh Recovery with Camera Consistency and Contrastive Lossescited by 0
  6. 2024Using Motion Cues to Supervise Single-Frame Body Pose and Shape Estimation in Low Data Regimescited by 0
  7. 2026RoSHI: A Versatile Robot-oriented Suit for Human Data In-the-Wildcited by 0