mlsanityData to 14 Oct 2026

SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control

arXiv 2605.22894 ↗First posted 21 May 2026cs.GR cs.LG cs.ROcited by 5

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Controlling physics-based humanoids from natural-language instructions is a critical step toward general-purpose embodied agents. However, existing methods remain constrained by a tension between semantic expressiveness and physical feasibility, often failing to jointly achieve faithful instruction following, high-quality motion, and stable long-horizon control. We propose SCRIPT, a scalable diffusion policy with a multi-stage training framework for language-driven physics-based humanoid control. The core of SCRIPT is a Joint Action-State-Text Diffusion Transformer (JAST-DiT), which represents actions, physical states, and text as dedicated token streams and couples them through joint attention, enabling direct interaction between language semantics and control dynamics. To stabilize autoregressive control, we introduce a nonlinear history conditioning mechanism, which preserves the dense recent context and samples increasingly sparse cues from long-term history. Beyond supervised imitation pre-training, we propose a post-training stage, further improving the performance using Reinforcement Learning with Hybrid Rewards (RLHR). By injecting learnable noise into the flow-sampling process, RLHR effectively improves motion quality and instruction following within closed-loop simulations using hybrid physical feedback and text rewards. Quantitative evaluations demonstrate that SCRIPT outperforms prior state-of-the-art methods, with gains across text alignment, motion quality, and physical realism metrics. Furthermore, scaling studies on the 1200-hour MotionMillion dataset demonstrate consistent performance gains with model scaling, highlighting SCRIPT's robust scalability for large-scale pre-training. Our code will be publicly available for future research.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 58 of its 91 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

58 of 91 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026World Action Models are Zero-shot Policies17 Feb
  2. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion11 Aug
  3. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  4. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards14 Mar
  5. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation28 Oct
  6. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control4 Oct
  7. 2024PDP: Physics-Based Character Animation via Diffusion Policy3 Jun
  8. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  9. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion4 Apr

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  8. 2025Qwen3-VL Technical Reportcited by 2,793
  9. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,236
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  11. 2022Human Motion Diffusion Modelcited by 1,080
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
37 more
  1. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  4. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  5. 2016The KIT Motion-Language Datasetcited by 338
  6. 2020Query-Key Normalization for Transformerscited by 280
  7. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  8. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  9. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
  10. 2023Object Motion Guided Human Motion Synthesiscited by 212
  11. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  12. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 178
  13. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
  14. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  15. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
  16. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 94
  17. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
  18. 2024Human-Object Interaction from Human-Level Instructionscited by 64
  19. 2025Embodied AI Agents: Modeling the Worldcited by 61
  20. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 60
  21. 2024MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understandingcited by 59
  22. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 53
  23. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 51
  24. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  25. 2023PADL: Language-Directed Physics-Based Character Controlcited by 48
  26. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  27. 2023Neural Categorical Priors for Physics-Based Character Controlcited by 38
  28. 2025MotionGPT3: Human Motion as a Second Modalitycited by 37
  29. 2026Kimodo: Scaling Controllable Human Motion Generationcited by 37
  30. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 25
  31. 2025VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulationcited by 25
  32. 2025Embody 3D: A Large-scale Multimodal Motion and Behavior Datasetcited by 19
  33. 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 17
  34. 2026Flow Policy Gradients for Robot Controlcited by 16
  35. 2025SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Controlcited by 15
  36. 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokenscited by 11
  37. 2024MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control ↗
  2. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance
  3. 2025Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary
  4. 2025RL from Physical Feedback: Aligning Large Motion Models with Humanoid Control
  5. 2025LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning
  6. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning
  7. 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructions
  8. 2024Text-Aware Diffusion for Policy Learning