# JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery

arXiv: https://arxiv.org/abs/2307.16377 · HTML: https://mlsanity.com/paper/2307.16377/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 90 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)

## Built on by (most cited first, 12 of 12)

- [SIFU: Side-view Conditioned Implicit Function for Real-world Usable Clothed Human Reconstruction](https://mlsanity.com/paper/2312.06704/) (2023)
- [RoHM: Robust Human Motion Reconstruction via Diffusion](https://mlsanity.com/paper/2401.08570/) (2024)
- [DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2404.01424/) (2024)
- [Alignment is All You Need: A Training-free Augmentation Strategy for Pose-guided Video Generation](https://mlsanity.com/paper/2408.16506/) (2024)
- [Distribution and Depth-Aware Transformers for 3D Human Mesh Recovery](https://mlsanity.com/paper/2403.09063/) (2024)
- [Multimodal Active Measurement for Human Mesh Recovery in Close Proximity](https://mlsanity.com/paper/2310.08116/) (2023)
- [Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction](https://mlsanity.com/paper/2510.26196/) (2025)
- [RUMPL: Ray-Based Transformers for Universal Multi-View 2D to 3D Human Pose Lifting](https://mlsanity.com/paper/2512.15488/) (2025)
- [Masked Modeling for Human Motion Recovery Under Occlusions](https://mlsanity.com/paper/2601.16079/) (2026)
- [TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction](https://mlsanity.com/paper/2604.08921/) (2026)
- [MoPO: Incorporating Motion Prior for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2605.09856/) (2026)
- [HumanFlow -- Diffusion-Driven MAV Navigation Among Humans via Tightly-Coupled Motion Tracking, Forecasting, and Control](https://mlsanity.com/paper/2605.25685/) (2026)

## Abstract (arXiv)

In this study, we focus on the problem of 3D human mesh recovery from a single image under obscured conditions. Most state-of-the-art methods aim to improve 2D alignment technologies, such as spatial averaging and 2D joint sampling. However, they tend to neglect the crucial aspect of 3D alignment by improving 3D representations. Furthermore, recent methods struggle to separate the target human from occlusion or background in crowded scenes as they optimize the 3D space of target human with 3D joint coordinates as local supervision. To address these issues, a desirable method would involve a framework for fusing 2D and 3D features and a strategy for optimizing the 3D space globally. Therefore, this paper presents 3D JOint contrastive learning with TRansformers (JOTR) framework for handling occluded 3D human mesh recovery. Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D&3D aligned results in a coarse-to-fine manner and a novel 3D joint contrastive learning approach for adding explicitly global supervision for the 3D feature space. The contrastive learning approach includes two contrastive losses: joint-to-joint contrast for enhancing the similarity of semantically similar voxels (i.e., human joints), and joint-to-non-joint contrast for ensuring discrimination from others (e.g., occlusions and background). Qualitative and quantitative analyses demonstrate that our method outperforms state-of-the-art competitors on both occlusion-specific and standard benchmarks, significantly improving the reconstruction of occluded humans.