# MagicAvatar: Multimodal Avatar Generation and Animation

arXiv: https://arxiv.org/abs/2308.14748 · HTML: https://mlsanity.com/paper/2308.14748/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 6 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 5 of 5)

- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)

## Built on by (most cited first, 17 of 17)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance](https://mlsanity.com/paper/2403.14781/) (2024)
- [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos](https://mlsanity.com/paper/2404.17571/) (2024)
- [TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models](https://mlsanity.com/paper/2403.17005/) (2024)
- [ViT-Lens: Towards Omni-modal Representations](https://mlsanity.com/paper/2311.16081/) (2023)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [XAGen: 3D Expressive Human Avatars Generation](https://mlsanity.com/paper/2311.13574/) (2023)
- [Towards Multiple Character Image Animation Through Enhancing Implicit Decoupling](https://mlsanity.com/paper/2406.03035/) (2024)
- [LoopAnimate: Loopable Salient Object Animation](https://mlsanity.com/paper/2404.09172/) (2024)
- [ViMo: Generating Motions from Casual Videos](https://mlsanity.com/paper/2408.06614/) (2024)
- [CFSynthesis: Controllable and Free-view 3D Human Video Synthesis](https://mlsanity.com/paper/2412.11067/) (2024)
- [Learning to Animate Images from A Few Videos to Portray Delicate Human Actions](https://mlsanity.com/paper/2503.00276/) (2025)
- [PaCo-FR: Patch-Pixel Aligned End-to-End Codebook Learning for Facial Representation Pre-training](https://mlsanity.com/paper/2508.09691/) (2025)
- [InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation](https://mlsanity.com/paper/2311.16499/) (2023)

## Abstract (arXiv)

This report presents MagicAvatar, a framework for multimodal video generation and animation of human avatars. Unlike most existing methods that generate avatar-centric videos directly from multimodal inputs (e.g., text prompts), MagicAvatar explicitly disentangles avatar video generation into two stages: (1) multimodal-to-motion and (2) motion-to-video generation. The first stage translates the multimodal inputs into motion/ control signals (e.g., human pose, depth, DensePose); while the second stage generates avatar-centric video guided by these motion signals. Additionally, MagicAvatar supports avatar animation by simply providing a few images of the target person. This capability enables the animation of the provided human identity according to the specific motion derived from the first stage. We demonstrate the flexibility of MagicAvatar through various applications, including text-guided and video-guided avatar generation, as well as multimodal avatar animation.