mlsanityData to 13 Oct 2026

TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning

arXiv 2605.12236 ↗First posted 12 May 2026cs.RO cs.AI cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

also in 1 more
Abstract (arXiv)

Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC), which produces narrow action distributions that lack the coverage necessary for downstream exploration. We present a unified framework that enables the exploration necessary to enable efficient robot policy finetuning by bridging BC pre-training and RL fine-tuning. Our pre-training method, Context-Smoothed Pre-training (CSP), injects forward-diffusion noise into policy inputs, creating a continuum between precise imitation and broad action coverage. We then fine-tune pre-trained policies via Timestep-Modulated Reinforcement Learning (TMRL), which trains the agent to dynamically adjust this conditioning during fine-tuning by modulating the diffusion timestep, granting explicit control over exploration. Integrating seamlessly with arbitrary policy inputs, e.g., states, 3D point clouds, or image-based VLA policies, we show that TMRL improves RL fine-tuning sample efficiency. Notably, TMRL enables successful real-world fine-tuning on complex manipulation tasks in under one hour. Videos and code available at https://weirdlabuw.github.io/tmrl/.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 68 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

59 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct

Other references · 58

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2020Denoising Diffusion Implicit Modelscited by 9,718
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  5. 2022Scalable Diffusion Models with Transformerscited by 5,973
  6. 2022Classifier-Free Diffusion Guidancecited by 5,785
  7. 2022Flow Matching for Generative Modelingcited by 5,542
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,077
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,760
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,344
  12. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,368
38 more
  1. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,336
  2. 2019Behavior Regularized Offline Reinforcement Learningcited by 657
  3. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 653
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 640
  5. 2021Implicit Behavioral Cloningcited by 442
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 334
  7. 2025π*0.6: a VLA That Learns From Experiencecited by 286
  8. 2024Diffusion Policy Policy Optimizationcited by 260
  9. 2024Reinforcement learningcited by 238
  10. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 229
  11. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  12. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 169
  13. 2025History-Guided Video Diffusioncited by 167
  14. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 165
  15. 2021A System for General In-Hand Object Re-Orientationcited by 155
  16. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 155
  17. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 155
  18. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 137
  19. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  20. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  21. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  22. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  23. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  24. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  25. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  26. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 50
  27. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 49
  28. 2025ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Modelscited by 43
  29. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  30. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 36
  31. 2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
  32. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 32
  33. 2025The Coverage Principle: How Pre-Training Enables Post-Trainingcited by 30
  34. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
  35. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  36. 2021Bundled Gradients through Contact via Randomized Smoothingcited by 23
  37. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  38. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Roboticscited by 4
  2. 2026Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experiencecited by 0
  3. 2026Imperfection for Precision: Upcycling Imperfect Data for High-Precision Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Imitation: Reinforcement Learning Fine-Tuning for Adaptive Diffusion Navigation Policies
  2. 2026MODIP: Efficient Model-Based Optimization for Diffusion Policies
  3. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning
  4. 2025Exploration and Adaptation in Non-Stationary Tasks with Diffusion Policies
  5. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
  6. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps
  7. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
  8. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models