# Under the Cover Infant Pose Estimation using Multimodal Data

arXiv: https://arxiv.org/abs/2210.00662 · HTML: https://mlsanity.com/paper/2210.00662/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [TransPose: Keypoint Localization via Transformer](https://mlsanity.com/paper/2012.14214/) (2020)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data](https://mlsanity.com/paper/2004.01166/) (2020)
- [Invariant Representation Learning for Infant Pose Estimation with Small Data](https://mlsanity.com/paper/2010.06100/) (2020)
- [BodyPressure -- Inferring Body Pose and Contact Pressure from a Depth Image](https://mlsanity.com/paper/2105.09936/) (2021)
- [Multimodal In-bed Pose and Shape Estimation under the Blankets](https://mlsanity.com/paper/2012.06735/) (2020)
- [3D Human Pose Estimation on a Configurable Bed from a Pressure Image](https://mlsanity.com/paper/1804.07873/) (2018)
- [AggPose: Deep Aggregation Vision Transformer for Infant Pose Estimation](https://mlsanity.com/paper/2205.05277/) (2022)
- [Towards human-level performance on automatic pose estimation of infant spontaneous movements](https://mlsanity.com/paper/2010.05949/) (2020)
- [Preterm infants' pose estimation with spatio-temporal features](https://mlsanity.com/paper/2005.08648/) (2020)
- [In-bed Pressure-based Pose Estimation using Image Space Representation Learning](https://mlsanity.com/paper/1908.08919/) (2019)
- [In-Bed Human Pose Estimation from Unseen and Privacy-Preserving Image Domains](https://mlsanity.com/paper/2111.15124/) (2021)
- [Towards Accurate Cross-Domain In-Bed Human Pose Estimation](https://mlsanity.com/paper/2110.03578/) (2021)
- [Privacy-Preserving In-Bed Pose Monitoring: A Fusion and Reconstruction Study](https://mlsanity.com/paper/2202.10704/) (2022)
- [Measuring uncertainty during respiratory rate estimation using pressure-sensitive mats](https://mlsanity.com/paper/1805.00082/) (2018)

## Built on by (most cited first, 6 of 6)

- [Keypoint-Integrated Instruction-Following Data Generation for Enhanced Human Pose and Action Understanding in Multimodal Models](https://mlsanity.com/paper/2409.09306/) (2024)
- [LLaVA-Pose: Enhancing Human Pose and Action Understanding via Keypoint-Integrated Instruction Tuning](https://mlsanity.com/paper/2506.21317/) (2025)
- [Self-Soupervision: Cooking Model Soups without Labels](https://mlsanity.com/paper/2602.02890/) (2026)
- [NurtureNet: A Multi-task Video-based Approach for Newborn Anthropometry](https://mlsanity.com/paper/2405.05530/) (2024)
- [Sleep Position Classification using Transfer Learning for Bed-based Pressure Sensors](https://mlsanity.com/paper/2505.08111/) (2025)
- [A Mixed Reality System for Robust Manikin Localization in Childbirth Training](https://mlsanity.com/paper/2602.05588/) (2026)

## Abstract (arXiv)

Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region of interest detection and movement detection for noncontact based monitoring systems. In a home setting, pose detection can be used to detect sleep positions which has shown to have a strong influence on multiple health factors. However, pose monitoring during sleep is challenging due to heavy occlusions from blanket coverings and low lighting. To address this, we present a novel dataset, Simultaneously-collected multimodal Mannequin Lying pose (SMaL) dataset, for under the cover infant pose estimation. We collect depth and pressure imagery of an infant mannequin in different poses under various cover conditions. We successfully infer full body pose under the cover by training state-of-art pose estimation methods and leveraging existing multimodal adult pose datasets for transfer learning. We demonstrate a hierarchical pretraining strategy for transformer-based models to significantly improve performance on our dataset. Our best performing model was able to detect joints under the cover within 25mm 86% of the time with an overall mean error of 16.9mm. Data, code and models publicly available at https://github.com/DanielKyr/SMaL