mlsanityData to 7 Oct 2026

Temporal Self-Imitation Learning

arXiv 2606.19752 ↗First posted 18 Jun 2026cs.RO cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Long-horizon policies trained with reinforcement learning can still achieve high return through inefficient interactions, while rare efficient behaviors discovered during training may be forgotten. We argue that temporal efficiency itself provides a source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 30 long-horizon tasks spanning robot manipulation and interactive navigation, TSIL consistently improves task success rates, learning efficiency, behavioral efficiency, and robustness to unstable training conditions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 39 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

30 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Self-Improving Embodied Foundation Models18 Sep
  2. 2023Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning11 Jun
  3. 2021Self-Imitation Learning by Planning25 Mar

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  5. 2018Self-Imitation Learningcited by 135
  6. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  7. 2017Time Limits in Reinforcement Learningcited by 72
  8. 2019Making Deep Q-learning methods robust to time discretizationcited by 51
  9. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  10. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
  11. 2019Real-Time Reinforcement Learningcited by 38
  12. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
15 more
  1. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  2. 2021TempoRL: Learning When to Actcited by 25
  3. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  4. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  5. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  6. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  7. 2025Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Taskscited by 10
  8. 2020Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewardscited by 4
  9. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  10. 2020Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertaintycited by 3
  11. 2021Reinforcement Learning for Robot Navigation with Adaptive Forward Simulation Time (AFST) in a Semi-Markov Modelcited by 2
  12. 2021Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learningcited by 1
  13. 2022STIR2: Reward Relabelling for combined Reinforcement and Imitation Learning on sparse-reward taskscited by 1
  14. 2024Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Drivingcited by 1
  15. 2025Time as a Control Dimension in Robot Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
  2. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learning
  3. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
  4. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation
  5. 2026AdaTempo: Learning Shared Relative Tempo from Demonstrations for Faster Robot Manipulation
  6. 2026SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulation
  7. 2026SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning
  8. 2026EgoSpeedUp: Transferring Human Manipulation Tempo to Robot Policies