# MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks

arXiv: https://arxiv.org/abs/2112.10082 · HTML: https://mlsanity.com/paper/2112.10082/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 69 references that are in the map.
- First posted on arXiv 19 Dec 2021.

## Builds on (most cited first, 30 of 43)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose](https://mlsanity.com/paper/2008.09047/) (2020)
- [Video Based Reconstruction of 3D People Models](https://mlsanity.com/paper/1803.04758/) (2018)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [Unsupervised 3D Pose Estimation with Geometric Self-Supervision](https://mlsanity.com/paper/1904.04812/) (2019)
- [Neural Kinematic Networks for Unsupervised Motion Retargetting](https://mlsanity.com/paper/1804.05653/) (2018)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [3D Human Pose Estimation in the Wild by Adversarial Learning](https://mlsanity.com/paper/1803.09722/) (2018)
- [CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild](https://mlsanity.com/paper/2011.14679/) (2020)

## Built on by (most cited first, 13 of 13)

- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [AthletePose3D: A Benchmark Dataset for 3D Human Pose Estimation and Kinematic Validation in Athletic Movements](https://mlsanity.com/paper/2503.07499/) (2025)
- [Deep learning for 3D human pose estimation and mesh recovery: A survey](https://mlsanity.com/paper/2402.18844/) (2024)
- [HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting](https://mlsanity.com/paper/2303.10941/) (2023)
- [Real-time Animation Generation and Control on Rigged Models via Large Language Models](https://mlsanity.com/paper/2310.17838/) (2023)
- [Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction](https://mlsanity.com/paper/2405.07444/) (2024)
- [An Identity-Preserved Framework for Human Motion Transfer](https://mlsanity.com/paper/2204.06862/) (2022)
- [Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting](https://mlsanity.com/paper/2209.01470/) (2022)
- [ReActor: Reinforcement Learning for Physics-Aware Motion Retargeting](https://mlsanity.com/paper/2605.06593/) (2026)
- [Redefining Data Pairing for Motion Retargeting Leveraging a Human Body Prior](https://mlsanity.com/paper/2409.13208/) (2024)
- [AnyAct: Towards Human Reenactment of Character Motion From Video](https://mlsanity.com/paper/2605.15497/) (2026)
- [ReFM: Semantic-Aware Refinement Flow Model for Motion Retargeting](https://mlsanity.com/paper/2609.32068/) (2026)

## Abstract (arXiv)

We present a novel framework that brings the 3D motion retargeting task from controlled environments to in-the-wild scenarios. In particular, our method is capable of retargeting body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure. It is designed to leverage massive online videos for unsupervised training, needless of 3D annotations or motion-body pairing information. The proposed method is built upon two novel canonicalization operations, structure canonicalization and view canonicalization. Trained with the canonicalization operations and the derived regularizations, our method learns to factorize a skeleton sequence into three independent semantic subspaces, i.e., motion, structure, and view angle. The disentangled representation enables motion retargeting from 2D to 3D with high precision. Our method achieves superior performance on motion transfer benchmarks with large body variations and challenging actions. Notably, the canonicalized skeleton sequence could serve as a disentangled and interpretable representation of human motion that benefits action analysis and motion retrieval.