mlsanityData to 14 Oct 2026

Reduced-Order Model-Guided Reinforcement Learning for Demonstration-Free Humanoid Locomotion

arXiv 2509.19023 ↗First posted 23 Sep 2025cs.RO cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

We introduce Reduced-Order Model-Guided Reinforcement Learning (ROM-GRL), a two-stage reinforcement learning framework for humanoid walking that requires no motion capture data or elaborate reward shaping. In the first stage, a compact 4-DOF (four-degree-of-freedom) reduced-order model (ROM) is trained via Proximal Policy Optimization. This generates energy-efficient gait templates. In the second stage, those dynamically consistent trajectories guide a full-body policy trained with Soft Actor--Critic augmented by an adversarial discriminator, ensuring the student's five-dimensional gait feature distribution matches the ROM's demonstrations. Experiments at 1 meter-per-second and 4 meter-per-second show that ROM-GRL produces stable, symmetric gaits with substantially lower tracking error than a pure-reward baseline. By distilling lightweight ROM guidance into high-dimensional policies, ROM-GRL bridges the gap between reward-only and imitation-based locomotion methods, enabling versatile, naturalistic humanoid behaviors without any human demonstrations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 24 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

12 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Gait-Conditioned Reinforcement Learning with Multi-Phase Curriculum for Humanoid Locomotion27 May
  2. 2024Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey25 Apr

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  4. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  5. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 94
  6. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  7. 2018Learning Symmetric and Low-energy Locomotioncited by 61
  8. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  9. 2020Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Modelcited by 15
  10. 2022Learning to Brachiate via Simplified Model Imitationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Gait Learning for Humanoid Robots Using Reinforcement Learning with Selective Adversarial Motion Prior
  2. 2026MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds
  3. 2025Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal
  4. 2024Opt2Skill: Imitating Dynamically-feasible Whole-Body Trajectories for Versatile Humanoid Loco-Manipulation
  5. 2026STRIDER: Stepping-Enabled Multi-Gait Hierarchical 3D Loco-Manipulation Framework for Humanoid Robots
  6. 2024Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion
  7. 2025RuN: Residual Policy for Natural Humanoid Locomotion
  8. 2026HuMBLE: Human Motion-Driven Behavior Learning for Embodied Locomotion ↗