mlsanityData to 14 Oct 2026

MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations

arXiv 2310.10198 ↗First posted 16 Oct 2023cs.CV cs.GRcited by 60

Not on the map. Why?

Abstract (arXiv)

In this work, we present MoConVQ, a novel unified framework for physics-based motion control leveraging scalable discrete representations. Building upon vector quantized variational autoencoders (VQ-VAE) and model-based reinforcement learning, our approach effectively learns motion embeddings from a large, unstructured dataset spanning tens of hours of motion examples. The resultant motion representation not only captures diverse motion skills but also offers a robust and intuitive interface for various applications. We demonstrate the versatility of MoConVQ through several applications: universal tracking control from various motion sources, interactive character control with latent motion representations using supervised learning, physics-based motion generation from natural language descriptions using the GPT framework, and, most interestingly, seamless integration with large language models (LLMs) with in-context learning to tackle complex and abstract tasks.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 61 of its 111 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

61 of 111 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
38 more
  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  4. 2022Human Motion Diffusion Modelcited by 1,080
  5. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  6. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  8. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  9. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
  10. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  11. 2020Jukebox: A Generative Model for Musiccited by 542
  12. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  13. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 501
  14. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
  15. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  16. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
  17. 2020HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimationcited by 307
  18. 2021Robust Motion In-betweeningcited by 229
  19. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  20. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
  21. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  22. 2021Character Controllers Using Motion VAEscited by 184
  23. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  24. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 163
  25. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 158
  26. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 114
  27. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 108
  28. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  29. 2023Synthesizing Physical Character-Scene Interactionscited by 104
  30. 2018Neural probabilistic motor primitives for humanoid controlcited by 97
  31. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 94
  32. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  33. 2020PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Timecited by 50
  34. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 49
  35. 2023PADL: Language-Directed Physics-Based Character Controlcited by 48
  36. 2019MoGlow: Probabilistic and controllable motion synthesis using normalising flowscited by 47
  37. 2023Human Pose as Compositional Tokenscited by 45
  38. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 40

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 24; 2026: 18.

60 papers.

Sort:

By area:

  1. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
  2. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 104
  3. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  4. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  5. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  6. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  7. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  8. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  9. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  10. 2024PDP: Physics-Based Character Animation via Diffusion Policycited by 31
  11. 2024MotionChain: Conversational Motion Controllers via Multimodal Promptscited by 30
  12. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
38 more
  1. 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Controlcited by 28
  2. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 25
  3. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  4. 2024LGTM: Local-to-Global Text-Driven Human Motion Diffusion Modelcited by 18
  5. 2024Pay Attention and Move Better: Harnessing Attention for Interactive Motion Generation and Training-free Editingcited by 18
  6. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 16
  7. 2025SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Controlcited by 15
  8. 2025Physics-Based Motion Imitation with Adversarial Differential Discriminatorscited by 12
  9. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  10. 2025Motion-example-controlled Co-speech Gesture Generation Leveraging Large Language Modelscited by 11
  11. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 11
  12. 2025Motion Generation: A Survey of Generative Approaches and Benchmarkscited by 8
  13. 2025Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputscited by 6
  14. 2025StableMotion: Training Motion Cleanup Models with Unpaired Corrupted Datacited by 6
  15. 2025Learning to Ball: Composing Policies for Long-Horizon Basketball Movescited by 6
  16. 2024LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesiscited by 5
  17. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 5
  18. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 5
  19. 2025CBIL: Collective Behavior Imitation Learning for Fish from Real Videoscited by 4
  20. 2025Moving by Looking: Towards Vision-Driven Avatar Motion Generationcited by 4
  21. 2026EasyTune: Efficient Step-Aware Fine-Tuning for Diffusion-Based Motion Generationcited by 4
  22. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
  23. 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic modelscited by 3
  24. 2025Versatile Physics-based Character Control with Hybrid Latent Representationcited by 3
  25. 2025Gen-C: Populating Virtual Worlds with Generative Crowdscited by 3
  26. 2026VQ-Style: Disentangling Style and Content in Motion with Residual Quantized Representationscited by 3
  27. 2024BID: Boundary-Interior Decoding for Unsupervised Temporal Action Localization Pre-Trainincited by 2
  28. 2025ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesiscited by 2
  29. 2025Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical Worldcited by 2
  30. 2025Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motionscited by 2
  31. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generationcited by 2
  32. 2026EchoAvatar: Real-time Generative Avatar Animation from Audio Streamscited by 2
  33. 2026GPC: Large-Scale Generative Pretraining for Transferable Motor Controlcited by 2
  34. 2026EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agentscited by 1
  35. 2025From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Controlcited by 0
  36. 2025Making Pose Representations More Expressive and Disentangled via Residual Vector Quantizationcited by 0
  37. 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenescited by 0
  38. 2025A Survey of Body and Face Motion: Datasets, Performance Evaluation Metrics and Generative Techniquescited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Re2MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinement
  2. 2026BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models ↗
  3. 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation
  4. 2026AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models
  5. 2025Physically-Grounded Goal Imagination: Physics-Informed Variational Autoencoder for Self-Supervised Reinforcement Learning
  6. 2026Motubrain: An Advanced World Action Model for Robot Control
  7. 2026ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation
  8. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Models