Sample, Simulate, Select: Physics-in-the-Loop Text-to-Motion for Humanoids Without Training
Not on the map. Why?
Abstract (arXiv)
Text-to-motion models generate plausible human motion but do not model a robot's dynamics; whole-body tracking controllers execute robot references reliably but cannot replan an infeasible one. Recent language-to-humanoid systems bridge this gap by training. We measure how much of the gap closes with no training at all, by putting the deployment controller itself in the loop. Sample-simulate-select (S3) draws N motions per prompt from a frozen text-to-motion model, retargets each to a Unitree G1 by direction-matching inverse kinematics, rolls all of them out under full rigid-body dynamics with the pretrained SONIC tracking policy, and keeps the candidate the policy executed best. Because the verifier is the deterministic simulator itself, S3 attains the any-of-N ceiling by construction; what we measure is where that ceiling lies and what falls short of it. On 200 stratified HumanML3D test prompts with N=8, upright execution rises from 83.5% to 89.5% and hardware-gate passes from 33 to 85; on the complete test split (4,184 prompts) it rises from 80.5% to 89.5%. A kinematic verifier that predicts falls well (AUROC 0.90) recovers only a quarter of this gain: ranking a prompt's own candidates is harder than classifying the population. What selection cannot fix is one class, prompts that lower the pelvis, which a generator trained on retargeted robot data does execute. We further score the semantic fidelity of the executed motion with the standard text-motion evaluator, with a real-mocap control that attributes the loss to the robot projection, ablate the retargeter against GMR (complementary failures: the any-of-8 ceiling rises to 95.0% over both), and execute all 177 gate-selected clips on the real G1: every one completes standing, with hardware tracking error matching simulation (r=0.94).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
37 of 41 references.
References · 37
- 2022Human Motion Diffusion Modelcited by 1,080
- 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
- 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
- 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
- 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 400
- 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 359
- 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 287
- 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 238
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
- 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 218
- 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 189
25 more
- 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 165
- 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 130
- 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 116
- 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 109
- 2025TWIST: Teleoperated Whole-Body Imitation Systemcited by 101
- 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 101
- 2024HOVER: Versatile Neural Whole-Body Controller for Humanoid Robotscited by 91
- 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 88
- 2025GMT: General Motion Tracking for Humanoid Whole-Body Controlcited by 83
- 2025Retargeting Matters: General Motion Retargeting for Humanoid Motion Trackingcited by 75
- 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
- 2025UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robotscited by 37
- 2026Kimodo: Scaling Controllable Human Motion Generationcited by 37
- 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 29
- 2025LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learningcited by 22
- 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 18
- 2026TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Controlcited by 14
- 2025RL from Physical Feedback: Aligning Large Motion Models with Humanoid Controlcited by 11
- 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructionscited by 10
- 2025Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabularycited by 7
- 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 5
- 2026ECHO: Edge-Cloud Humanoid Orchestration for Language-to-Motion Controlcited by 4
- 2026SafeFlow: Real-Time Text-Driven Humanoid Whole-Body Control via Physics-Guided Rectified Flow and Selective Safety Gatingcited by 3
- 2026TEXEDO : Test Time Scaling for Controller-aware Language-conditioned Humanoid Motion Generationcited by 3
- 2026PhyGile: Physics-Prefix Guided Motion Generation for Agile General Humanoid Motion Trackingcited by 2
Built on by
Not cited yet.