# Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision

arXiv: https://arxiv.org/abs/2503.06089 · HTML: https://mlsanity.com/paper/2503.06089/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera](https://mlsanity.com/paper/1803.05959/) (2018)
- [UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture](https://mlsanity.com/paper/2208.01633/) (2022)
- [SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera](https://mlsanity.com/paper/2011.01519/) (2020)
- [4D Human Body Capture from Egocentric Video via 3D Scene Grounding](https://mlsanity.com/paper/2011.13341/) (2020)
- [EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2304.07803/) (2023)
- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [Egocentric Pose Estimation from Human Vision Span](https://mlsanity.com/paper/2104.05167/) (2021)
- [3D Human Pose Perception from Egocentric Stereo Videos](https://mlsanity.com/paper/2401.00889/) (2023)
- [Egocentric Gesture Recognition for Head-Mounted AR devices](https://mlsanity.com/paper/1808.05380/) (2018)
- [Head and eye egocentric gesture recognition for human-robot interaction using eyewear cameras](https://mlsanity.com/paper/2201.11500/) (2022)

## Built on by (most cited first, 3 of 3)

- [Egocentric Whole-Body Human Mesh Recovery with Prior-Guided Learning](https://mlsanity.com/paper/2605.08606/) (2026)
- [Human Motion Estimation with Everyday Wearables](https://mlsanity.com/paper/2512.21209/) (2025)
- [AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimation](https://mlsanity.com/paper/2603.25175/) (2026)

## Abstract (arXiv)

Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's first-person perspective. Although research has used pose estimation techniques to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We introduce Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and 2D joints as auxiliary loss to support model training. To address the scarcity of egocentric camera data, we create a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms previous state-of-the-art 3D HMR models.