mlsanityData to 14 Oct 2026

PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control

arXiv 2606.21925 ↗First posted 20 Jun 2026math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we develop an off-policy method for continuous-time reinforcement learning (CTRL), where the system dynamics are governed by an unknown stochastic differential equation (SDE) and only discrete-time trajectory data are available. A central challenge is that the classical state-action value function Q(s,a), which enables off-policy learning in discrete-time RL, does not exist in CTRL (Baird, 1994; Jia and Zhou, 2023; Tallec et al., 2019). On the other hand, continuous-time control provides local notions such as the instantaneous advantage function q(s,a), but these typically rely on state value function V(s). To address this, we introduce a new definition of the state-action value function in CTRL and derive its governing equation. Building on the PhiBE approximation (Zhu, 2024; Zhu et al., 2025), we propose iterative algorithms to approximate the optimal Q-function in both model-based and model-free settings using only discrete-time off-policy data. Under linear function approximation, we establish convergence guarantees and derive explicit convergence rates for the proposed method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 32 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

16 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Optimal-PhiBE: A PDE-based Model-free framework for Continuous-time Reinforcement Learning5 Jun
  2. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation20 May
  3. 2025The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective27 Jan
  4. 2024PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation21 May

Other references · 12

  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 127
  5. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  6. 2022q-Learning in Continuous Timecited by 91
  7. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  8. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  9. 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
  10. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 23
  11. 2019Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Timecited by 9
  12. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  2. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions
  3. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
  4. 2023Boosting Continuous Control with Consistency Policy
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  7. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  8. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning