# FLAG: Flow-based 3D Avatar Generation from Sparse Observations

arXiv: https://arxiv.org/abs/2203.05789 · HTML: https://mlsanity.com/paper/2203.05789/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [Probabilistic Modeling for Human Mesh Recovery](https://mlsanity.com/paper/2108.11944/) (2021)
- [Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors](https://mlsanity.com/paper/2103.17265/) (2021)
- [We are More than Our Joints: Predicting how 3D Bodies Move](https://mlsanity.com/paper/2012.00619/) (2020)
- [Contact and Human Dynamics from Monocular Video](https://mlsanity.com/paper/2007.11678/) (2020)
- [Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows](https://mlsanity.com/paper/2003.10350/) (2020)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Monocular 3D Human Pose Estimation by Generation and Ordinal Ranking](https://mlsanity.com/paper/1904.01324/) (2019)
- [Neural Descent for Visual 3D Human Pose and Shape](https://mlsanity.com/paper/2008.06910/) (2020)
- [3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data](https://mlsanity.com/paper/2011.00980/) (2020)
- [THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers](https://mlsanity.com/paper/2106.09336/) (2021)
- [LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals](https://mlsanity.com/paper/2103.01500/) (2021)
- [Probabilistic 3D Human Shape and Pose Estimation from Multiple Unconstrained Images in the Wild](https://mlsanity.com/paper/2103.10978/) (2021)

## Built on by (most cited first, 30 of 55)

- [MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis](https://mlsanity.com/paper/2212.04495/) (2022)
- [Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model](https://mlsanity.com/paper/2304.08577/) (2023)
- [Ego-Body Pose Estimation via Ego-Head Pose Estimation](https://mlsanity.com/paper/2212.04636/) (2022)
- [QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars](https://mlsanity.com/paper/2209.09391/) (2022)
- [ReGenNet: Towards Human Action-Reaction Synthesis](https://mlsanity.com/paper/2403.11882/) (2024)
- [BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis](https://mlsanity.com/paper/2304.11118/) (2023)
- [Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling](https://mlsanity.com/paper/2308.08855/) (2023)
- [EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere](https://mlsanity.com/paper/2308.06493/) (2023)
- [Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera](https://mlsanity.com/paper/2401.00847/) (2024)
- [HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations](https://mlsanity.com/paper/2403.03561/) (2024)
- [HuManiFlow: Ancestor-Conditioned Normalising Flows on SO(3) Manifolds for Human Pose and Shape Distribution Estimation](https://mlsanity.com/paper/2305.06968/) (2023)
- [Look Ma, no markers: holistic performance capture without the hassle](https://mlsanity.com/paper/2410.11520/) (2024)
- [SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data](https://mlsanity.com/paper/2311.02191/) (2023)
- [Stratified Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2405.20786/) (2024)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations](https://mlsanity.com/paper/2308.11261/) (2023)
- [Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture](https://mlsanity.com/paper/2309.00310/) (2023)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals](https://mlsanity.com/paper/2404.04890/) (2024)
- [HERO: Human Reaction Generation from Videos](https://mlsanity.com/paper/2503.08270/) (2025)
- [SimpleEgo: Predicting Probabilistic Body Pose from Egocentric Cameras](https://mlsanity.com/paper/2401.14785/) (2024)
- [DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers](https://mlsanity.com/paper/2402.09211/) (2024)
- [EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling](https://mlsanity.com/paper/2412.10235/) (2024)
- [Physics-based Motion Retargeting from Sparse Inputs](https://mlsanity.com/paper/2307.01938/) (2023)
- [MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generation](https://mlsanity.com/paper/2311.14494/) (2023)
- [A Survey on 3D Egocentric Human Pose Estimation](https://mlsanity.com/paper/2403.17893/) (2024)
- [ARFlow: Human Action-Reaction Flow Matching with Physical Guidance](https://mlsanity.com/paper/2503.16973/) (2025)
- [MOVIN: Real-time Motion Capture using a Single LiDAR](https://mlsanity.com/paper/2309.09314/) (2023)
- [ET3D: Efficient Text-to-3D Generation via Multi-View Distillation](https://mlsanity.com/paper/2311.15561/) (2023)
- [Generalizable Face Landmarking Guided by Conditional Face Warping](https://mlsanity.com/paper/2404.12322/) (2024)

## Abstract (arXiv)

To represent people in mixed reality applications for collaboration and communication, we need to generate realistic and faithful avatar poses. However, the signal streams that can be applied for this task from head-mounted devices (HMDs) are typically limited to head pose and hand pose estimates. While these signals are valuable, they are an incomplete representation of the human body, making it challenging to generate a faithful full-body avatar. We address this challenge by developing a flow-based generative model of the 3D human body from sparse observations, wherein we learn not only a conditional distribution of 3D human pose, but also a probabilistic mapping from observations to the latent space from which we can generate a plausible pose along with uncertainty estimates for the joints. We show that our approach is not only a strong predictive model, but can also act as an efficient pose prior in different optimization settings where a good initial latent code plays a major role.