mlsanityData to 7 Oct 2026

TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

arXiv 2410.09536 ↗First posted 12 Oct 2024cs.LG cs.ROcited by 18

Not on the map. Why?

Abstract (arXiv)

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 36 of its 67 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

36 of 67 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
24 more
  1. 2019Differentiable Convex Optimization Layerscited by 458
  2. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  3. 2022Online Decision Transformercited by 194
  4. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  5. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  7. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  8. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  9. 2023Elastic Decision Transformercited by 66
  10. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  11. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
  12. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 45
  13. 2022ProDMPs: A Unified Perspective on Dynamic and Probabilistic Movement Primitivescited by 25
  14. 2021Specializing Versatile Skill Libraries using Local Mixture of Expertscited by 24
  15. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spacescited by 23
  16. 2022Deep Black-Box Reinforcement Learning with Movement Primitivescited by 18
  17. 2020High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewardscited by 14
  18. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  19. 2024Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learningcited by 12
  20. 2021Orientation Probabilistic Movement Primitives on Riemannian Manifoldscited by 10
  21. 2024Rethinking Transformers in Solving POMDPscited by 9
  22. 2023MP3: Movement Primitive-Based (Re-)Planning Policycited by 7
  23. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  24. 2024Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025Reinforcement Learning with Action Chunkingcited by 97
  2. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  3. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  4. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 34
  5. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 25
  6. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  7. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  8. 2025OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulationcited by 7
  9. 2026SEAR: Sample Efficient Action Chunking Reinforcement Learningcited by 5
  10. 2026Adaptive Action Chunking via Multi-Chunk Q Value Estimationcited by 4
  11. 2026Learning Visual Feature-Based World Models via Residual Latent Actioncited by 2
  12. 2025ReaCritic: Reasoning Transformer-based DRL Critic-model Scaling For Wireless Networkscited by 1
6 more
  1. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learningcited by 1
  2. 2025Accelerating Transformers in Online RLcited by 0
  3. 2026Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllerscited by 0
  4. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  5. 2026PAWS: Preference Learning with Advantage-Weighted Segmentscited by 0
  6. 2026Action Chunking Proximal Policy Optimization with Feedback Correctioncited by 0