mlsanityData to 14 Oct 2026

FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control

arXiv 2505.22642 ↗First posted 28 May 2025cs.RO cs.AI cs.LGcited by 44

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Reinforcement learning (RL) has driven significant progress in robotics, but its complexity and long training times remain major bottlenecks. In this report, we introduce FastTD3, a simple, fast, and capable RL algorithm that significantly speeds up training for humanoid robots in popular suites such as HumanoidBench, IsaacLab, and MuJoCo Playground. Our recipe is remarkably simple: we train an off-policy TD3 agent with several modifications -- parallel simulation, large-batch updates, a distributional critic, and carefully tuned hyperparameters. FastTD3 solves a range of HumanoidBench tasks in under 3 hours on a single A100 GPU, while remaining stable during training. We also provide a lightweight and easy-to-use implementation of FastTD3 to accelerate RL research in robotics.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 39 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

28 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Control24 Feb

Other references · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  10. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  12. 2019Learning agile and dynamic motor skills for legged robotscited by 504
15 more
  1. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  2. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  3. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  4. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  5. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  6. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  7. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  8. 2024Simplifying Deep Temporal Difference Learningcited by 78
  9. 2025MuJoCo Playgroundcited by 78
  10. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 76
  11. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  12. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  13. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 41
  14. 2024BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmarkcited by 30
  15. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 32.

44 papers.

Sort:

By area:

  1. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  2. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
  3. 2026Flow Policy Gradients for Robot Controlcited by 16
  4. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 15
  5. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  6. 2025Learning Massively Multitask World Models for Continuous Controlcited by 12
  7. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 8
  8. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  9. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  10. 2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4
  11. 2025OMP: One-step Meanflow Policy with Directional Alignmentcited by 4
  12. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 4
32 more
  1. 2026AGILE: A Comprehensive Workflow for Humanoid Loco-Manipulation Learningcited by 4
  2. 2026Combined Constrained Sampling and Reinforcement Learning for Robotic Manipulationcited by 3
  3. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 3
  4. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  5. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  6. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 2
  7. 2026Scaling Behavior Foundation Model for Humanoid Robotscited by 2
  8. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  9. 2026Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Controlcited by 1
  10. 2026Reward-Conditioned Reinforcement Learningcited by 1
  11. 2026NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learningcited by 1
  12. 2026FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Controlcited by 1
  13. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  14. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  15. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  16. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  17. 2026AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidancecited by 1
  18. 2025Functional Critics Are Essential for Actor-Critic: From Off-Policy Stability to Efficient Explorationcited by 0
  19. 2026MetaWorld-X: Hierarchical World Modeling via VLM-Orchestrated Experts for Humanoid Loco-Manipulationcited by 0
  20. 2026Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEscited by 0
  21. 2026Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learningcited by 0
  22. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?cited by 0
  23. 2026Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotioncited by 0
  24. 2026FlashNav: Training Deployable Robot Navigation Policies in Secondscited by 0
  25. 2026ReFPO: Reflow Regularization for Flow Matching Policy Gradientscited by 0
  26. 2026FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotioncited by 0
  27. 2026Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RLcited by 0
  28. 2026FERPO: Forward Entropy-Regularized Policy Optimizationcited by 0
  29. 2026Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimizationcited by 0
  30. 2026A Bird's-Eye View of Iterative Reward Designcited by 0
  31. 2026ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Modelscited by 0
  32. 2026Efficient Test-Time Adaptation For Robot Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robot Trains Robot: Automatic Real-World Policy Adaptation and Learning for Humanoids
  2. 2025Booster Gym: An End-to-End Reinforcement Learning Framework for Humanoid Robot Locomotion
  3. 2026QF3: Fast Flow RL with Filtered Q-Gradients
  4. 2025LiPS: Large-Scale Humanoid Robot Reinforcement Learning with Parallel-Series Structures
  5. 2026FastStair: Learning to Run Up Stairs with Humanoid Robots ↗
  6. 2024Data-Efficient Approach to Humanoid Control via Fine-Tuning a Pre-Trained GPT on Action Data
  7. 2025Mechanical Intelligence-Aware Curriculum Reinforcement Learning for Humanoids with Parallel Actuation
  8. 2026FetchMan: Learning Visual Humanoid Loco-Manipulation Policies from Simulated Experiences