# PromptHMR: Promptable Human Mesh Recovery

arXiv: https://arxiv.org/abs/2504.06397 · HTML: https://mlsanity.com/paper/2504.06397/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 71 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 30 of 66)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion](https://mlsanity.com/paper/2306.16940/) (2023)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [Monocular, One-stage, Regression of Multiple 3D People](https://mlsanity.com/paper/2008.12272/) (2020)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [AGORA: Avatars in Geography Optimized for Regression Analysis](https://mlsanity.com/paper/2104.14643/) (2021)

## Built on by (most cited first, 30 of 76)

- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [Human3R: Everyone Everywhere All at Once](https://mlsanity.com/paper/2510.06219/) (2025)
- [BEDLAM2.0: Synthetic Humans and Cameras in Motion](https://mlsanity.com/paper/2511.14394/) (2025)
- [Viser: Imperative, Web-based 3D Visualization in Python](https://mlsanity.com/paper/2507.22885/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Control](https://mlsanity.com/paper/2602.07439/) (2026)
- [SCAIL: Towards Studio-Grade Character Animation via In-Context Learning of 3D-Consistent Pose Representations](https://mlsanity.com/paper/2512.05905/) (2025)
- [UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collections](https://mlsanity.com/paper/2509.24817/) (2025)
- [SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos](https://mlsanity.com/paper/2512.08406/) (2025)
- [GentleHumanoid: Learning Upper-body Compliance for Contact-rich Human and Object Interaction](https://mlsanity.com/paper/2511.04679/) (2025)
- [DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction](https://mlsanity.com/paper/2603.03265/) (2026)
- [MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction](https://mlsanity.com/paper/2602.15733/) (2026)
- [SOMA: Unifying Parametric Human Body Models](https://mlsanity.com/paper/2603.16858/) (2026)
- [MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images](https://mlsanity.com/paper/2506.09919/) (2025)
- [Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2603.15603/) (2026)
- [Syn4D: A Multiview Synthetic 4D Dataset](https://mlsanity.com/paper/2605.05207/) (2026)
- [Group Inertial Poser: Multi-Person Pose and Global Translation from Sparse Inertial Sensors and Ultra-Wideband Ranging](https://mlsanity.com/paper/2510.21654/) (2025)
- [Towards Metric-Aware Multi-Person Mesh Recovery by Jointly Optimizing Human Crowd in Camera Space](https://mlsanity.com/paper/2511.13282/) (2025)
- [PEAR: Pixel-aligned Expressive humAn mesh Recovery](https://mlsanity.com/paper/2601.22693/) (2026)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [HumanScore: Benchmarking Human Motions in Generated Videos](https://mlsanity.com/paper/2604.20157/) (2026)
- [Learnable SMPLify: A Neural Solution for Optimization-Free Human Pose Inverse Kinematics](https://mlsanity.com/paper/2508.13562/) (2025)
- [Affordance-Guided Diffusion Prior for 3D Hand Reconstruction](https://mlsanity.com/paper/2510.00506/) (2025)
- [VLM-Guided Group Preference Alignment for Diffusion-based Human Mesh Recovery](https://mlsanity.com/paper/2602.19180/) (2026)
- [Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports](https://mlsanity.com/paper/2603.09896/) (2026)
- [From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching](https://mlsanity.com/paper/2603.26938/) (2026)
- [OmniFit: Multi-modal 3D Body Fitting via Scale-agnostic Dense Landmark Prediction](https://mlsanity.com/paper/2604.21575/) (2026)
- [SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild](https://mlsanity.com/paper/2605.07604/) (2026)
- [Multi-HMR 2: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking](https://mlsanity.com/paper/2606.14841/) (2026)
- [HumanMoveVQA: Can Video MLLMs reason about human movement in videos?](https://mlsanity.com/paper/2606.27999/) (2026)

## Abstract (arXiv)

Human pose and shape (HPS) estimation presents challenges in diverse scenarios such as crowded scenes, person-person interactions, and single-view reconstruction. Existing approaches lack mechanisms to incorporate auxiliary "side information" that could enhance reconstruction accuracy in such challenging scenarios. Furthermore, the most accurate methods rely on cropped person detections and cannot exploit scene context while methods that process the whole image often fail to detect people and are less accurate than methods that use crops. While recent language-based methods explore HPS reasoning through large language or vision-language models, their metric accuracy is well below the state of the art. In contrast, we present PromptHMR, a transformer-based promptable method that reformulates HPS estimation through spatial and semantic prompts. Our method processes full images to maintain scene context and accepts multiple input modalities: spatial prompts like bounding boxes and masks, and semantic prompts like language descriptions or interaction labels. PromptHMR demonstrates robust performance across challenging scenarios: estimating people from bounding boxes as small as faces in crowded scenes, improving body shape estimation through language descriptions, modeling person-person interactions, and producing temporally coherent motions in videos. Experiments on benchmarks show that PromptHMR achieves state-of-the-art performance while offering flexible prompt-based control over the HPS estimation process.