mlsanityData to 7 Oct 2026

PADL: Language-Directed Physics-Based Character Control

arXiv 2301.13868 ↗First posted 31 Jan 2023cs.LG cs.AI cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Developing systems that can synthesize natural and life-like motions for simulated characters has long been a focus for computer animation. But in order for these systems to be useful for downstream applications, they need not only produce high-quality motions, but must also provide an accessible and versatile interface through which users can direct a character's behaviors. Natural language provides a simple-to-use and expressive medium for specifying a user's intent. Recent breakthroughs in natural language processing (NLP) have demonstrated effective use of language-based interfaces for applications such as image generation and program synthesis. In this work, we present PADL, which leverages recent innovations in NLP in order to take steps towards developing language-directed controllers for physics-based character animation. PADL allows users to issue natural language commands for specifying both high-level tasks and low-level skills that a character should perform. We present an adversarial imitation learning approach for training policies to map high-level language commands to low-level controls that enable a character to perform the desired task and skill specified by a user's commands. Furthermore, we propose a multi-task aggregation method that leverages a language-based multiple-choice question-answering approach to determine high-level task objectives from language commands. We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 31 of its 72 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

31 of 72 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
19 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  3. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  4. 2015Policy Distillationcited by 414
  5. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  6. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  7. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  8. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  9. 2021Character Controllers Using Motion VAEscited by 182
  10. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  11. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  12. 2017Learning human behaviors from motion capture by adversarial imitationcited by 75
  13. 2017Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networkscited by 61
  14. 2017Divide-and-Conquer Reinforcement Learningcited by 46
  15. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  16. 2021Neural Video Compression using GANs for Detail Synthesis and Propagationcited by 35
  17. 2022ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characterscited by 6
  18. 2021Learning to Manipulate Amorphous Materialscited by 5
  19. 2018Text2Scene: Generating Compositional Scenes from Textual Descriptionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 13; 2025: 10; 2026: 9.

47 papers.

Sort:

By area:

  1. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  2. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  3. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  4. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  5. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  6. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 91
  7. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  8. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  9. 2023C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characterscited by 53
  10. 2023Physically Plausible Full-Body Hand-Object Interaction Synthesiscited by 52
  11. 2023Synthesizing Physically Plausible Human Motions in 3D Scenescited by 47
  12. 2024CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamicscited by 43
35 more
  1. 2023Iterative Motion Editing with Natural Languagecited by 39
  2. 2023Neural Categorical Priors for Physics-Based Character Controlcited by 37
  3. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  4. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  5. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 25
  6. 2024PACER+: On-Demand Pedestrian Animation Controller in Driving Scenarioscited by 24
  7. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  8. 2024PhysReaction: Physically Plausible Real-Time Humanoid Reaction Synthesis via Forward Dynamics Guided 4D Imitationcited by 18
  9. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 17
  10. 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 16
  11. 2023AdaptNet: Policy Adaptation for Physics-Based Character Controlcited by 15
  12. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  13. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  14. 2024SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generationcited by 12
  15. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  16. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  17. 2025Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputscited by 6
  18. 2025FineXtrol: Controllable Motion Generation via Fine-Grained Textcited by 4
  19. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 4
  20. 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Sizecited by 4
  21. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 4
  22. 2024GPT-Connect: Interaction between Text-Driven Human Motion Generator and 3D Scenes in a Training-free Mannercited by 3
  23. 2024ChatDyn: Language-Driven Multi-Actor Dynamics Generation in Street Scenescited by 3
  24. 2023MAAIP: Multi-Agent Adversarial Interaction Priors for imitation from fighting demonstrations for physics-based characterscited by 2
  25. 2025Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motionscited by 2
  26. 2024PlaMo: Plan and Move in Rich 3D Physical Environmentscited by 1
  27. 2025ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulationcited by 1
  28. 2026Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid Controlcited by 1
  29. 2026MIND: Multi-Scale Intent Diffusion for Text-Driven Physics-Based Humanoid Controlcited by 1
  30. 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenescited by 0
  31. 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcingcited by 0
  32. 2026In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physicscited by 0
  33. 2026Progressively Learning Heterogeneous Skills in a Unified Latent Spacecited by 0
  34. 2026LYRIC: Language-Driven Physics-Based Character Control for Contact-Rich Whole-Body Object Interactioncited by 0
  35. 2026Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023InsActor: Instruction-driven Physics-based Characters
  2. 2023Synthesizing Physical Character-Scene Interactions
  3. 2021A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
  4. 2024PDP: Physics-Based Character Animation via Diffusion Policy
  5. 2021AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control
  6. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks
  7. 2026Language-Critique Imitation Learning from Suboptimal Demonstrations
  8. 2020Language Conditioned Imitation Learning over Unstructured Data