mlsanityData to 7 Oct 2026

Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning

arXiv 2401.11437 ↗First posted 21 Jan 2024cs.LG cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

Current advancements in reinforcement learning (RL) have predominantly focused on learning step-based policies that generate actions for each perceived state. While these methods efficiently leverage step information from environmental interaction, they often ignore the temporal correlation between actions, resulting in inefficient exploration and unsmooth trajectories that are challenging to implement on real hardware. Episodic RL (ERL) seeks to overcome these challenges by exploring in parameters space that capture the correlation of actions. However, these approaches typically compromise data efficiency, as they treat trajectories as opaque black boxes. In this work, we introduce a novel ERL algorithm, Temporally-Correlated Episodic RL (TCE), which effectively utilizes step information in episodic policy updates, opening the 'black box' in existing ERL methods while retaining the smooth and consistent exploration in parameter space. TCE synergistically combines the advantages of step-based and episodic RL, achieving comparable performance to recent ERL methods while maintaining data efficiency akin to state-of-the-art (SoTA) step-based RL.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 25 of its 61 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

25 of 61 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  10. 2019Differentiable Convex Optimization Layerscited by 458
  11. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
  12. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 45
13 more
  1. 2022DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systemscited by 27
  2. 2021TempoRL: Learning When to Actcited by 25
  3. 2022ProDMPs: A Unified Perspective on Dynamic and Probabilistic Movement Primitivescited by 25
  4. 2021Specializing Versatile Skill Libraries using Local Mixture of Expertscited by 24
  5. 2023Latent Exploration for Reinforcement Learningcited by 21
  6. 2022Deep Black-Box Reinforcement Learning with Movement Primitivescited by 18
  7. 2021TAAC: Temporally Abstract Actor-Critic for Continuous Controlcited by 14
  8. 2021Jerk-limited Real-time Trajectory Generation with Arbitrary Target Statescited by 12
  9. 2021Orientation Probabilistic Movement Primitives on Riemannian Manifoldscited by 10
  10. 2023MP3: Movement Primitive-Based (Re-)Planning Policycited by 7
  11. 2016Policy Search with High-Dimensional Context Variablescited by 5
  12. 2023Deep Probabilistic Movement Primitives with a Bayesian Aggregatorcited by 4
  13. 2016A Probabilistic Representation for Dynamic Movement Primitivescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 2.

12 papers.

  1. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 25
  2. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
  3. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  4. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  5. 2025Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objectscited by 10
  6. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Modelscited by 7
  7. 2024Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learningcited by 4
  8. 2025Context-aware Learned Mesh-based Simulation via Trajectory-Level Meta-Learningcited by 1
  9. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Controlcited by 0
  10. 2025MoRe-ERL: Learning Motion Residuals using Episodic Reinforcement Learningcited by 0
  11. 2026PAWS: Preference Learning with Advantage-Weighted Segmentscited by 0
  12. 2026Trust-Region Diffusion Policies for Massively Parallel On-Policy RLcited by 0