mlsanityData to 7 Oct 2026

Multi-step Reinforcement Learning: A Unifying Algorithm

arXiv 1703.01327 ↗First posted 3 Mar 2017cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD(λ) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter λ. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, Q-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called Q(σ) which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, σ, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). Q(σ) is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of σ, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 7; 2024: 10; 2025: 6; 2026: 7.

42 papers.

Sort:

By area:

  1. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  2. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  3. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  4. 2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
  5. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  6. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  7. 2024CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Drivingcited by 12
  8. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  9. 2021Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineeringcited by 6
  10. 2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
  11. 2023Provably Convergent Policy Optimization via Metric-aware Trust Region Methodscited by 6
  12. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
30 more
  1. 2025BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learningcited by 4
  2. 2023Multi-Bellman operator for convergence of Q-learning with linear function approximationcited by 3
  3. 2024Highway Reinforcement Learningcited by 2
  4. 2024Reflective Policy Optimizationcited by 2
  5. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  6. 2021Policies for the Dynamic Traveling Maintainer Problem with Alertscited by 1
  7. 2024Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximationcited by 1
  8. 2024Two-Step Q-Learningcited by 1
  9. 2026Adaptive Scaffolding for Cognitive Engagement in an Intelligent Tutoring Systemcited by 1
  10. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  11. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 1
  12. 2021Greedy-Step Off-Policy Reinforcement Learningcited by 0
  13. 2021Learning to Plan via a Multi-Step Policy Regression Methodcited by 0
  14. 2022Adaptive Tree Backup Algorithms for Temporal-Difference Reinforcement Learningcited by 0
  15. 2022Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observabilitycited by 0
  16. 2022Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problemscited by 0
  17. 2023On the Benefits of Leveraging Structural Information in Planning Over the Learned Modelcited by 0
  18. 2023Learned spatial data partitioningcited by 0
  19. 2023Towards Optimal Pricing of Demand Response -- A Nonparametric Constrained Policy Optimization Approachcited by 0
  20. 2023Bias Resilient Multi-Step Off-Policy Goal-Conditioned Reinforcement Learningcited by 0
  21. 2024Actor-Critic Reinforcement Learning with Phased Actorcited by 0
  22. 2024Multi-State TD Target for Model-Free Reinforcement Learningcited by 0
  23. 2024A Multi-Step Minimax Q-learning Algorithm for Two-Player Zero-Sum Markov Gamescited by 0
  24. 2025Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximationcited by 0
  25. 2025Time After Time: Deep-Q Effect Estimation for Interventions on When and What to docited by 0
  26. 2025Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learningcited by 0
  27. 2026Soft Q(λ): A multi-step off-policy method for entropy regularised reinforcement learning using eligibility tracescited by 0
  28. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalitiescited by 0
  29. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  30. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learningcited by 0