mlsanityData to 7 Oct 2026

Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generations

arXiv 2412.13111 ↗First posted 17 Dec 2024cs.CV cs.GRcited by 9

Not on the map. Why?

Abstract (arXiv)

Text-driven human motion synthesis has showcased its potential for revolutionizing motion design in the movie and game industry. Existing methods often rely on 3D motion capture data, which requires special setups, resulting in high costs for data acquisition, ultimately limiting the diversity and scope of human motion. In contrast, 2D human videos offer a vast and accessible source of motion data, covering a wider range of styles and activities. In this paper, we explore the use of 2D human motion extracted from videos as an alternative data source to improve text-driven 3D motion generation. Our approach introduces a novel framework that disentangles local joint motion from global movements, enabling efficient learning of local motion priors from 2D data. We first train a single-view 2D local motion generator on a large dataset of text-2D motion pairs. Then we fine-tune the generator with 3D data, transforming it into a multi-view generator that predicts view-consistent local joint motion and root dynamics. Evaluations on the well-acknowledged dataset and novel text prompts demonstrate that our method can efficiently utilize 2D data, supporting a wider range of realistic 3D human motion generation. Our code is publicly available at https://zju3dv.github.io/Motion-2-to-3/.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 79 of its 90 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

79 of 90 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  10. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  11. 2022Human Motion Diffusion Modelcited by 1,062
  12. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  13. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  14. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  15. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  16. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  17. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  18. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  19. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  20. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  21. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  22. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  23. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  24. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  25. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  26. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  27. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  28. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  29. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  30. 2023Human Motion Diffusion as a Generative Priorcited by 354
  31. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  32. 2016The KIT Motion-Language Datasetcited by 334
  33. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  34. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  35. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  36. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  37. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  38. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  2. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  3. 2026CoMoVi: Co-Generation of 3D Human Motions and Realistic Videoscited by 4
  4. 2025Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motionscited by 2
  5. 2025Mocap-2-to-3: Multi-view Lifting for Monocular Motion Recovery with 2D Pretrainingcited by 0
  6. 2025LaxMotion: Rethinking Supervision Granularity for 3D Human Motion Generationcited by 0
  7. 2026Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascadescited by 0
  8. 2026Unified Number-Free Text-to-Motion Generation Via Flow Matchingcited by 0
  9. 2026VideoMDM: Towards 3D Human Motion Generation From 2D Supervisioncited by 0