mlsanityData to 14 Oct 2026

Intentional Updates for Streaming Reinforcement Learning

arXiv 2604.19033 ↗First posted 21 Apr 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In gradient-based learning, a step size chosen in parameter units does not produce a predictable per-step change in function output. This often leads to instability in the streaming setting (i.e., batch size=1), where stochasticity is not averaged out and update magnitudes can momentarily become arbitrarily big or small. Instead, we propose intentional updates: first specify the intended outcome of an update and then solve for the step size that approximately achieves it. This strategy has precedent in online supervised linear regression via Normalized Least Mean Squares algorithm, which selects a step size to yield a specified change in the function output proportional to the current error. We extend this principle to streaming deep reinforcement learning by defining appropriate intended outcomes: Intentional TD aims for a fixed fractional reduction of the TD error, and Intentional Policy Gradient aims for a bounded per-step change in the policy, limiting local KL divergence. We propose practical algorithms combining eligibility traces and diagonal scaling. Empirically, these methods yield state-of-the-art streaming performance, frequently performing on par with batch and replay-buffer approaches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 26 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

12 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Streaming Deep Reinforcement Learning Finally Works18 Oct

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  7. 2017Time Limits in Reinforcement Learningcited by 79
  8. 2020Expected Eligibility Tracescited by 27
  9. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 18
  10. 2015True Online Temporal-Difference Learningcited by 16
  11. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning
  2. 2026Revisiting Adam for Streaming Reinforcement Learning
  3. 2026Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning
  4. 2026Scalable Reinforcement Learning via Adaptive Batch Scaling
  5. 2026An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics
  6. 2026Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
  7. 2025Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning
  8. 2026SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL