# Neural Categorical Priors for Physics-Based Character Control

arXiv: https://arxiv.org/abs/2308.07200 · HTML: https://mlsanity.com/paper/2308.07200/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 88 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies](https://mlsanity.com/paper/1905.09808/) (2019)
- [Synthesizing Physical Character-Scene Interactions](https://mlsanity.com/paper/2302.00883/) (2023)
- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Learning Symmetric and Low-energy Locomotion](https://mlsanity.com/paper/1801.08093/) (2018)
- [Theory and Experiments on Vector Quantized Autoencoders](https://mlsanity.com/paper/1805.11063/) (2018)
- [ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills](https://mlsanity.com/paper/2005.04323/) (2020)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)
- [MoGlow: Probabilistic and controllable motion synthesis using normalising flows](https://mlsanity.com/paper/1905.06598/) (2019)
- [ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters](https://mlsanity.com/paper/2210.06063/) (2022)
- [TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game](https://mlsanity.com/paper/2011.13729/) (2020)
- [Animating Human Athletics](https://mlsanity.com/paper/2302.06108/) (2023)
- [Discovering Diverse Athletic Jumping Strategies](https://mlsanity.com/paper/2105.00371/) (2021)
- [TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2011.12895/) (2020)

## Built on by (most cited first, 30 of 38)

- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization](https://mlsanity.com/paper/2503.19901/) (2025)
- [Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models](https://mlsanity.com/paper/2308.15143/) (2023)
- [PDP: Physics-Based Character Animation via Diffusion Policy](https://mlsanity.com/paper/2406.00960/) (2024)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [SMPLOlympics: Sports Environments for Physically Simulated Humanoids](https://mlsanity.com/paper/2407.00187/) (2024)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control](https://mlsanity.com/paper/2503.11801/) (2025)
- [Strategy and Skill Learning for Physics-based Table Tennis Animation](https://mlsanity.com/paper/2407.16210/) (2024)
- [SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control](https://mlsanity.com/paper/2512.03028/) (2025)
- [CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects](https://mlsanity.com/paper/2505.21437/) (2025)
- [Physics-based Scene Layout Generation from Human Motion](https://mlsanity.com/paper/2405.12460/) (2024)
- [Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models](https://mlsanity.com/paper/2408.11801/) (2024)
- [AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2505.23708/) (2025)
- [ModSkill: Physical Character Skill Modularization](https://mlsanity.com/paper/2502.14140/) (2025)
- [Learning to Ball: Composing Policies for Long-Horizon Basketball Moves](https://mlsanity.com/paper/2509.22442/) (2025)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)
- [SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control](https://mlsanity.com/paper/2605.22894/) (2026)
- [Versatile Physics-based Character Control with Hybrid Latent Representation](https://mlsanity.com/paper/2503.12814/) (2025)
- [Learning Human-Like Badminton Skills for Humanoid Robots](https://mlsanity.com/paper/2602.08370/) (2026)
- [An Efficient Model-Based Approach on Learning Agile Motor Skills without Reinforcement](https://mlsanity.com/paper/2403.01962/) (2024)
- [Feature-Based vs. GAN-Based Learning from Demonstrations: When and Why](https://mlsanity.com/paper/2507.05906/) (2025)
- [GPC: Large-Scale Generative Pretraining for Transferable Motor Control](https://mlsanity.com/paper/2606.29148/) (2026)
- [ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling](https://mlsanity.com/paper/2410.06963/) (2024)
- [Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Control](https://mlsanity.com/paper/2602.21599/) (2026)
- [MUSIC: Learning Muscle-Driven Dexterous Hand Control](https://mlsanity.com/paper/2604.23886/) (2026)
- [HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction](https://mlsanity.com/paper/2608.16222/) (2026)
- [Integrating Controllable Motion Skills from Demonstrations](https://mlsanity.com/paper/2408.03018/) (2024)

## Abstract (arXiv)

Recent advances in learning reusable motion priors have demonstrated their effectiveness in generating naturalistic behaviors. In this paper, we propose a new learning framework in this paradigm for controlling physics-based characters with significantly improved motion quality and diversity over existing state-of-the-art methods. The proposed method uses reinforcement learning (RL) to initially track and imitate life-like movements from unstructured motion clips using the discrete information bottleneck, as adopted in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure compresses the most relevant information from the motion clips into a compact yet informative latent space, i.e., a discrete space over vector quantized codes. By sampling codes in the space from a trained categorical prior distribution, high-quality life-like behaviors can be generated, similar to the usage of VQ-VAE in computer vision. Although this prior distribution can be trained with the supervision of the encoder's output, it follows the original motion clip distribution in the dataset and could lead to imbalanced behaviors in our setting. To address the issue, we further propose a technique named prior shifting to adjust the prior distribution using curiosity-driven RL. The outcome distribution is demonstrated to offer sufficient behavioral diversity and significantly facilitates upper-level policy learning for downstream tasks. We conduct comprehensive experiments using humanoid characters on two challenging downstream tasks, sword-shield striking and two-player boxing game. Our results demonstrate that the proposed framework is capable of controlling the character to perform considerably high-quality movements in terms of behavioral strategies, diversity, and realism. Videos, codes, and data are available at https://tencent-roboticsx.github.io/NCP/.