mlsanityData to 7 Oct 2026

CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimation

arXiv 2407.01244 ↗First posted 1 Jul 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

In the monocular setting, predicting 3D pose and shape of animals typically relies solely on visual information, which is highly under-constrained. In this work, we explore using audio to enhance 3D shape and motion recovery of horses from monocular video. We test our approach on two datasets: an indoor treadmill dataset for 3D evaluation and an outdoor dataset capturing diverse horse movements, the latter being a contribution to this study. Our results show that incorporating sound with visual data leads to more accurate and robust motion regression. This study is the first to investigate audio's role in 3D animal motion recovery.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 49 of its 67 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

49 of 67 references.

References · 49

  1. 2017End-to-end Recovery of Human Shape and Posecited by 883
  2. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  3. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  4. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  5. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  6. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  7. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  8. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  9. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  10. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  11. 2018Learning 3D Human Dynamics from Videocited by 315
  12. 20163D Menagerie: Modeling the 3D shape and pose of animalscited by 310
37 more
  1. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  2. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  3. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  4. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  5. 2016One-Shot Video Object Segmentationcited by 233
  6. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  7. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  8. 2018Markerless tracking of user-defined features with deep learningcited by 203
  9. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  10. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  11. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  12. 2018Learning Factorized Multimodal Representationscited by 171
  13. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  14. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  15. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  16. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 134
  17. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  18. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  19. 2019Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images "In the Wild"cited by 106
  20. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  21. 2021Learning Motion Priors for 4D Human Body Capture in 3D Scenescited by 98
  22. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  23. 2018Creatures great and SMAL: Recovering the shape and motion of animals from videocited by 84
  24. 2020Synthetic Training for Accurate 3D Human Pose and Shape Estimation in the Wildcited by 82
  25. 2020Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loopcited by 81
  26. 2019Analyzing Input and Output Representations for Speech-Driven Gesture Generationcited by 51
  27. 2022BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Informationcited by 51
  28. 2017Audio to Body Dynamicscited by 48
  29. 20203D Bird Reconstruction: a Dataset, Model, and Shape Recovery from a Single Viewcited by 40
  30. 2020RGBD-Dog: Predicting Canine Pose from RGBD Sensorscited by 35
  31. 2021Birds of a Feather: Capturing Avian Shape Models from Imagescited by 30
  32. 2020Beyond Weak Perspective for Monocular 3D Human Pose Estimationcited by 29
  33. 2020Speech Prediction in Silent Videos using Variational Autoencoderscited by 18
  34. 2021hSMAL: Detailed Horse Shape and Pose Reconstruction for Motion Pattern Recognitioncited by 18
  35. 2020Chained Representation Cycling: Learning to Estimate 3D Human Pose and Shape by Cycling Between Representationscited by 12
  36. 2019EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddingscited by 8
  37. 2021PoseKernelLifter: Metric Lifting of 3D Human Pose using Soundcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Towards Multi-Modal Animal Pose Estimation: A Survey and In-Depth Analysiscited by 2