# PADL: Language-Directed Physics-Based Character Control

arXiv: https://arxiv.org/abs/2301.13868 · HTML: https://mlsanity.com/paper/2301.13868/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 72 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Character Controllers Using Motion VAEs](https://mlsanity.com/paper/2103.14274/) (2021)
- [DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills](https://mlsanity.com/paper/1804.02717/) (2018)
- [SimPoE: Simulated Character Control for 3D Human Pose Estimation](https://mlsanity.com/paper/2104.00683/) (2021)
- [Learning human behaviors from motion capture by adversarial imitation](https://mlsanity.com/paper/1707.02201/) (2017)
- [Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks](https://mlsanity.com/paper/1705.06400/) (2017)
- [Divide-and-Conquer Reinforcement Learning](https://mlsanity.com/paper/1711.09874/) (2017)
- [UniCon: Universal Neural Controller For Physics-based Character Motion](https://mlsanity.com/paper/2011.15119/) (2020)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters](https://mlsanity.com/paper/2205.01906/) (2022)
- [Learning to Manipulate Amorphous Materials](https://mlsanity.com/paper/2103.02533/) (2021)

## Built on by (most cited first, 30 of 48)

- [OmniControl: Control Any Joint at Any Time for Human Motion Generation](https://mlsanity.com/paper/2310.08580/) (2023)
- [MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting](https://mlsanity.com/paper/2409.14393/) (2024)
- [Synthesizing Diverse Human Motions in 3D Indoor Scenes](https://mlsanity.com/paper/2305.12411/) (2023)
- [CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control](https://mlsanity.com/paper/2410.03441/) (2024)
- [CALM: Conditional Adversarial Latent Models for Directable Virtual Characters](https://mlsanity.com/paper/2305.02195/) (2023)
- [Unified Human-Scene Interaction via Prompted Chain-of-Contacts](https://mlsanity.com/paper/2309.07918/) (2023)
- [TLControl: Trajectory and Language Control for Human Motion Synthesis](https://mlsanity.com/paper/2311.17135/) (2023)
- [MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations](https://mlsanity.com/paper/2310.10198/) (2023)
- [C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters](https://mlsanity.com/paper/2309.11351/) (2023)
- [Physically Plausible Full-Body Hand-Object Interaction Synthesis](https://mlsanity.com/paper/2309.07907/) (2023)
- [Synthesizing Physically Plausible Human Motions in 3D Scenes](https://mlsanity.com/paper/2308.09036/) (2023)
- [CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics](https://mlsanity.com/paper/2406.14558/) (2024)
- [Iterative Motion Editing with Natural Language](https://mlsanity.com/paper/2312.11538/) (2023)
- [Neural Categorical Priors for Physics-Based Character Control](https://mlsanity.com/paper/2308.07200/) (2023)
- [AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents](https://mlsanity.com/paper/2403.12835/) (2024)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [SkillMimic: Learning Basketball Interaction Skills from Demonstrations](https://mlsanity.com/paper/2408.15270/) (2024)
- [UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Control](https://mlsanity.com/paper/2504.12540/) (2025)
- [PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarios](https://mlsanity.com/paper/2404.19722/) (2024)
- [Generalized Animal Imitator: Agile Locomotion with Versatile Motion Prior](https://mlsanity.com/paper/2310.01408/) (2023)
- [PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitation](https://mlsanity.com/paper/2404.01081/) (2024)
- [SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation](https://mlsanity.com/paper/2407.10481/) (2024)
- [AdaptNet: Policy Adaptation for Physics-Based Character Control](https://mlsanity.com/paper/2310.00239/) (2023)
- [Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking](https://mlsanity.com/paper/2412.17730/) (2024)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation](https://mlsanity.com/paper/2411.19921/) (2024)
- [GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skill](https://mlsanity.com/paper/2504.04191/) (2025)
- [ModSkill: Physical Character Skill Modularization](https://mlsanity.com/paper/2502.14140/) (2025)
- [Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs](https://mlsanity.com/paper/2502.05641/) (2025)
- [InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphs](https://mlsanity.com/paper/2512.07410/) (2025)

## Abstract (arXiv)

Developing systems that can synthesize natural and life-like motions for simulated characters has long been a focus for computer animation. But in order for these systems to be useful for downstream applications, they need not only produce high-quality motions, but must also provide an accessible and versatile interface through which users can direct a character's behaviors. Natural language provides a simple-to-use and expressive medium for specifying a user's intent. Recent breakthroughs in natural language processing (NLP) have demonstrated effective use of language-based interfaces for applications such as image generation and program synthesis. In this work, we present PADL, which leverages recent innovations in NLP in order to take steps towards developing language-directed controllers for physics-based character animation. PADL allows users to issue natural language commands for specifying both high-level tasks and low-level skills that a character should perform. We present an adversarial imitation learning approach for training policies to map high-level language commands to low-level controls that enable a character to perform the desired task and skill specified by a user's commands. Furthermore, we propose a multi-task aggregation method that leverages a language-based multiple-choice question-answering approach to determine high-level task objectives from language commands. We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.