mlsanityData to 14 Oct 2026

WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

arXiv 2609.29106 ↗First posted 24 Sep 2026cs.CV cs.AI cs.CGcited by 0

Not on the map. Why?

Abstract (arXiv)

3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine-tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference the head predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person identity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin-aware Sinkhorn association of per-frame bodies. WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video. Each window is predicted feed-forward; analytic association and Sim(3) composition connect windows. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

38 of 43 references.

References · 38

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  3. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  4. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  5. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  8. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  9. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  10. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  11. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 350
  12. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
26 more
  1. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 206
  2. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  3. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  4. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 153
  5. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  6. 2024World-Grounded Human Motion Recovery via Gravity-View Coordinatescited by 137
  7. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 106
  8. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 101
  9. 2023EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wildcited by 97
  10. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 91
  11. 2025Easi3R: Estimating Disentangled Motion from DUSt3R Without Trainingcited by 91
  12. 2025PromptHMR: Promptable Human Mesh Recoverycited by 76
  13. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  14. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  15. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 34
  16. 2024Reconstructing People, Places, and Camerascited by 31
  17. 2025Joint Optimization for 4D Human-Scene Reconstruction in the Wildcited by 29
  18. 2025AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backendcited by 20
  19. 2024COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimationcited by 15
  20. 2025HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstructioncited by 15
  21. 2020People as Scene Probescited by 14
  22. 2026MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sourcescited by 10
  23. 2024Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recoverycited by 4
  24. 2025MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Imagescited by 4
  25. 2026Scene and Human in One World: Reconstruction in a Feedforward Passcited by 2
  26. 2026GUSH3R: Everyone Everywhere All at Once as Gaussianscited by 1

Built on by

Not cited yet.