mlsanityData to 14 Oct 2026

Generalized Fitted Q-Iteration with Clustered Data

arXiv 2510.03912 ↗First posted 4 Oct 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra-cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q-function and policy estimators when the correlation structure is correctly specified, and (ii) their consistencies when the structure is mis-specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 51 of its 109 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

51 of 109 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data14 May

Other references · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  9. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  10. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  11. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  12. 2017Real-Time Bidding by Reinforcement Learning in Display Advertisingcited by 91
38 more
  1. 2020FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Financecited by 87
  2. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  3. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  4. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  5. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  6. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  7. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  8. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  9. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  10. 2021A Deep Value-network Based Approach for Multi-Driver Order Dispatchingcited by 45
  11. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  12. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  13. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  14. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  15. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  16. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  17. 2021Optimal policy evaluation using kernel-based temporal difference methodscited by 24
  18. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  19. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  20. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  21. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  22. 2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learningcited by 8
  23. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
  24. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  25. 2020Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policycited by 7
  26. 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
  27. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  28. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
  29. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  30. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  31. 2023Randomization Inference When N Equals Onecited by 5
  32. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  33. 2025Deep Distributional Learning with Non-crossing Quantile Networkcited by 5
  34. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  35. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  36. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  37. 2025Semi-pessimistic Reinforcement Learningcited by 2
  38. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders
  2. 2026Finite-Sample Theory for Fitted Q-Iteration When Actions Are Functions
  3. 2024Fitted Q-Iteration via Max-Plus-Linear Approximation
  4. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  5. 2025Harnessing Data from Clustered LQR Systems: Personalized and Collaborative Policy Optimization
  6. 2025Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization
  7. 2025A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning
  8. 2021Approximated Multi-Agent Fitted Q Iteration