mlsanityData to 14 Oct 2026

Is Q-learning an Ill-posed Problem?

arXiv 2502.14365 ↗First posted 20 Feb 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper investigates the instability of Q-learning in continuous environments, a challenge frequently encountered by practitioners. Traditionally, this instability is attributed to bootstrapping and regression model errors. Using a representative reinforcement learning benchmark, we systematically examine the effects of bootstrapping and model inaccuracies by incrementally eliminating these potential error sources. Our findings reveal that even in relatively simple benchmarks, the fundamental task of Q-learning - iteratively learning a Q-function from policy-specific target values - can be inherently ill-posed and prone to failure. These insights cast doubt on the reliability of Q-learning as a universal solution for reinforcement learning problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 3 of its 6 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

3 of 6 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Why long model-based rollouts are no reason for bad Q-value estimates16 Jul

Other references · 2

  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  2. 2023Fractal Landscapes in Policy Optimizationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  2. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  3. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  4. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  5. 2026Implicit Q-Learning and SARSA: Liberating Policy Control from Step-Size Calibration
  6. 2026Understanding and Stabilizing Deep Q-Learning via Controlled Bootstrapping and Regulated Value Dynamics
  7. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
  8. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning