# Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction

arXiv: https://arxiv.org/abs/2103.16449 · HTML: https://mlsanity.com/paper/2103.16449/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 65 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision](https://mlsanity.com/paper/1611.09813/) (2016)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.03713/) (2020)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Episodic Training for Domain Generalization](https://mlsanity.com/paper/1902.00113/) (2019)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [Unite the People: Closing the Loop Between 3D and 2D Human Representations](https://mlsanity.com/paper/1701.02468/) (2017)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithms](https://mlsanity.com/paper/1908.10400/) (2019)
- [Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation](https://mlsanity.com/paper/2004.03686/) (2020)
- [Exploiting temporal context for 3D human pose estimation in the wild](https://mlsanity.com/paper/1905.04266/) (2019)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019)

## Built on by (most cited first, 30 of 39)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images](https://mlsanity.com/paper/2207.06400/) (2022)
- [Recovering 3D Human Mesh from Monocular Images: A Survey](https://mlsanity.com/paper/2203.01923/) (2022)
- [3D Human Pose Estimation via Intuitive Physics](https://mlsanity.com/paper/2303.18246/) (2023)
- [Playing for 3D Human Recovery](https://mlsanity.com/paper/2110.07588/) (2021)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [Occluded Human Mesh Recovery](https://mlsanity.com/paper/2203.13349/) (2022)
- [GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation](https://mlsanity.com/paper/2207.09763/) (2022)
- [AdaptPose: Cross-Dataset Adaptation for 3D Human Pose Estimation by Learnable Motion Generation](https://mlsanity.com/paper/2112.11593/) (2021)
- [MeshMAE: Masked Autoencoders for 3D Mesh Data Analysis](https://mlsanity.com/paper/2207.10228/) (2022)
- [Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions](https://mlsanity.com/paper/2410.20294/) (2024)
- [PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos](https://mlsanity.com/paper/2404.04430/) (2024)
- [Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation](https://mlsanity.com/paper/2203.15293/) (2022)
- [Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation](https://mlsanity.com/paper/2111.04017/) (2021)
- [Global Adaptation meets Local Generalization: Unsupervised Domain Adaptation for 3D Human Pose Estimation](https://mlsanity.com/paper/2303.16456/) (2023)
- [ZeroAvatar: Zero-shot 3D Avatar Generation from a Single Image](https://mlsanity.com/paper/2305.16411/) (2023)
- [Cyclic Test-Time Adaptation on Monocular Video for 3D Human Mesh Reconstruction](https://mlsanity.com/paper/2308.06554/) (2023)
- [Domain Adaptive 3D Pose Augmentation for In-the-wild Human Mesh Recovery](https://mlsanity.com/paper/2206.10457/) (2022)
- [JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2307.16377/) (2023)
- [PoSynDA: Multi-Hypothesis Pose Synthesis Domain Adaptation for Robust 3D Human Pose Estimation](https://mlsanity.com/paper/2308.09678/) (2023)
- [Lifting 2D Human Pose to 3D with Domain Adapted 3D Body Concept](https://mlsanity.com/paper/2111.11969/) (2021)
- [Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening](https://mlsanity.com/paper/2210.04294/) (2022)
- [MoCap-to-Visual Domain Adaptation for Efficient Human Mesh Estimation from 2D Keypoints](https://mlsanity.com/paper/2404.07094/) (2024)
- [NeMo: 3D Neural Motion Fields from Multiple Video Instances of the Same Action](https://mlsanity.com/paper/2212.13660/) (2022)
- [STRIDE: Single-video based Temporally Continuous Occlusion-Robust 3D Pose Estimation](https://mlsanity.com/paper/2312.16221/) (2023)
- [Uncertainty-Aware Testing-Time Optimization for 3D Human Pose Estimation](https://mlsanity.com/paper/2402.02339/) (2024)
- [Personalized 3D Human Pose and Shape Refinement](https://mlsanity.com/paper/2403.11634/) (2024)
- [Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery](https://mlsanity.com/paper/2401.14121/) (2024)
- [ESCAPE: Energy-based Selective Adaptive Correction for Out-of-distribution 3D Human Pose Estimation](https://mlsanity.com/paper/2407.14605/) (2024)

## Abstract (arXiv)

This paper considers a new problem of adapting a pre-trained model of human mesh reconstruction to out-of-domain streaming videos. However, most previous methods based on the parametric SMPL model citeloper2015smpl underperform in new domains with unexpected, domain-specific attributes, such as camera parameters, lengths of bones, backgrounds, and occlusions. Our general idea is to dynamically fine-tune the source model on test video streams with additional temporal constraints, such that it can mitigate the domain gaps without over-fitting the 2D information of individual test frames. A subsequent challenge is how to avoid conflicts between the 2D and temporal constraints. We propose to tackle this problem using a new training algorithm named Bilevel Online Adaptation (BOA), which divides the optimization process of overall multi-objective into two steps of weight probe and weight update in a training iteration. We demonstrate that BOA leads to state-of-the-art results on two human mesh reconstruction benchmarks.