SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control
On the map: Diffusion models → Diffusion and flow policies for robotics
Abstract (arXiv)
Controlling physics-based humanoids from natural-language instructions is a critical step toward general-purpose embodied agents. However, existing methods remain constrained by a tension between semantic expressiveness and physical feasibility, often failing to jointly achieve faithful instruction following, high-quality motion, and stable long-horizon control. We propose SCRIPT, a scalable diffusion policy with a multi-stage training framework for language-driven physics-based humanoid control. The core of SCRIPT is a Joint Action-State-Text Diffusion Transformer (JAST-DiT), which represents actions, physical states, and text as dedicated token streams and couples them through joint attention, enabling direct interaction between language semantics and control dynamics. To stabilize autoregressive control, we introduce a nonlinear history conditioning mechanism, which preserves the dense recent context and samples increasingly sparse cues from long-term history. Beyond supervised imitation pre-training, we propose a post-training stage, further improving the performance using Reinforcement Learning with Hybrid Rewards (RLHR). By injecting learnable noise into the flow-sampling process, RLHR effectively improves motion quality and instruction following within closed-loop simulations using hybrid physical feedback and text rewards. Quantitative evaluations demonstrate that SCRIPT outperforms prior state-of-the-art methods, with gains across text alignment, motion quality, and physical realism metrics. Furthermore, scaling studies on the 1200-hour MotionMillion dataset demonstrate consistent performance gains with model scaling, highlighting SCRIPT's robust scalability for large-scale pre-training. Our code will be publicly available for future research.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 58 of its 91 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
58 of 91 references.
Closest ancestors · 9
Same direction, newest first
- 2026World Action Models are Zero-shot Policies17 Feb
- 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion11 Aug
- 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
- 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards14 Mar
- 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation28 Oct
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control4 Oct
- 2024PDP: Physics-Based Character Animation via Diffusion Policy3 Jun
- 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
- 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion4 Apr
Other references · 49
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
- 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
- 2022Human Motion Diffusion Modelcited by 1,080
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
37 more
- 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
- 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
- 2016The KIT Motion-Language Datasetcited by 338
- 2020Query-Key Normalization for Transformerscited by 280
- 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
- 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
- 2023Object Motion Guided Human Motion Synthesiscited by 212
- 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
- 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
- 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
- 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
- 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
- 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 94
- 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
- 2024Human-Object Interaction from Human-Level Instructionscited by 64
- 2025Embodied AI Agents: Modeling the Worldcited by 61
- 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 60
- 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
- 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
- 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
- 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
- 2023PADL: Language-Directed Physics-Based Character Controlcited by 48
- 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
- 2023Neural Categorical Priors for Physics-Based Character Controlcited by 38
- 2025MotionGPT3: Human Motion as a Second Modalitycited by 37
- 2026Kimodo: Scaling Controllable Human Motion Generationcited by 37
- 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 25
- 2025VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulationcited by 25
- 2025Embody 3D: A Large-scale Multimodal Motion and Behavior Datasetcited by 19
- 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 17
- 2026Flow Policy Gradients for Robot Controlcited by 16
- 2025SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Controlcited by 15
- 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokenscited by 11
- 2024MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learningcited by 10
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026GenTrack: Physical Alignment for Robot-Native Motion Generation and Zero-Shot Humanoid Trackingcited by 2
- 2026Human-Centric Intelligence in the Era of Foundation Models: A Surveycited by 0
- 2026LYRIC: Language-Driven Physics-Based Character Control for Contact-Rich Whole-Body Object Interactioncited by 0
- 2026PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Controlcited by 0
- 2026Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Trainingcited by 0
- 2026Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training22 Sep 2026
- 2026PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control21 Sep 2026
- 2026LYRIC: Language-Driven Physics-Based Character Control for Contact-Rich Whole-Body Object Interaction17 Sep 2026
- 2026Human-Centric Intelligence in the Era of Foundation Models: A Survey18 Aug 2026
- 2026GenTrack: Physical Alignment for Robot-Native Motion Generation and Zero-Shot Humanoid Tracking2 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control ↗
- 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance
- 2025Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary
- 2025RL from Physical Feedback: Aligning Large Motion Models with Humanoid Control
- 2025LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning
- 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning
- 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructions
- 2024Text-Aware Diffusion for Policy Learning