# Multi-modal 3D Pose and Shape Estimation with Computed Tomography

arXiv: https://arxiv.org/abs/2503.19405 · HTML: https://mlsanity.com/paper/2503.19405/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Accurate 3D Body Shape Regression using Metric and Semantic Attributes](https://mlsanity.com/paper/2206.07036/) (2022)
- [AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild](https://mlsanity.com/paper/2010.13302/) (2020)
- [Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data](https://mlsanity.com/paper/2004.01166/) (2020)
- [BodyPressure -- Inferring Body Pose and Contact Pressure from a Depth Image](https://mlsanity.com/paper/2105.09936/) (2021)
- [Seeing Under the Cover: A Physics Guided Learning Approach for In-Bed Pose Estimation](https://mlsanity.com/paper/1907.02161/) (2019)
- [Multimodal In-bed Pose and Shape Estimation under the Blankets](https://mlsanity.com/paper/2012.06735/) (2020)
- [3D Human Pose Estimation on a Configurable Bed from a Pressure Image](https://mlsanity.com/paper/1804.07873/) (2018)
- [RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation](https://mlsanity.com/paper/1907.00135/) (2019)
- [Anatomy-guided domain adaptation for 3D in-bed human pose estimation](https://mlsanity.com/paper/2211.12193/) (2022)
- [Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds](https://mlsanity.com/paper/2107.07539/) (2021)
- [In-bed Pressure-based Pose Estimation using Image Space Representation Learning](https://mlsanity.com/paper/1908.08919/) (2019)
- [BodyMAP -- Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed](https://mlsanity.com/paper/2404.03183/) (2024)
- [Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion](https://mlsanity.com/paper/2403.03217/) (2024)

## Abstract (arXiv)

In perioperative care, precise in-bed 3D patient pose and shape estimation (PSE) can be vital in optimizing patient positioning in preoperative planning, enabling accurate overlay of medical images for augmented reality-based surgical navigation, and mitigating risks of prolonged immobility during recovery. Conventional PSE methods relying on modalities such as RGB-D, infrared, or pressure maps often struggle with occlusions caused by bedding and complex patient positioning, leading to inaccurate estimation that can affect clinical outcomes. To address these challenges, we present the first multi-modal in-bed patient 3D PSE network that fuses detailed geometric features extracted from routinely acquired computed tomography (CT) scans with depth maps (mPSE-CT). mPSE-CT incorporates a shape estimation module that utilizes probabilistic correspondence alignment, a pose estimation module with a refined neural network, and a final parameters mixing module. This multi-modal network robustly reconstructs occluded body regions and enhances the accuracy of the estimated 3D human mesh model. We validated mPSE-CT using proprietary whole-body rigid phantom and volunteer datasets in clinical scenarios. mPSE-CT outperformed the best-performing prior method by 23% and 49.16% in pose and shape estimation respectively, demonstrating its potential for improving clinical outcomes in challenging perioperative environments.