# Generalized Fitted Q-Iteration with Clustered Data

arXiv: https://arxiv.org/abs/2510.03912 · HTML: https://mlsanity.com/paper/2510.03912/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 51 of its 109 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 51)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Real-Time Bidding by Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1701.02490/) (2017)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [Finite-Sample Analysis for SARSA with Linear Function Approximation](https://mlsanity.com/paper/1902.02234/) (2019)
- [Settling the Sample Complexity of Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.05275/) (2022)
- [Batch Policy Learning in Average Reward Markov Decision Processes](https://mlsanity.com/paper/2007.11771/) (2020)
- [Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning](https://mlsanity.com/paper/1611.03531/) (2016)
- [Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising](https://mlsanity.com/paper/1802.09756/) (2018)
- [Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings](https://mlsanity.com/paper/2001.04515/) (2020)
- [CoinDICE: Off-Policy Confidence Interval Estimation](https://mlsanity.com/paper/2010.11652/) (2020)
- [Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency](https://mlsanity.com/paper/2102.02981/) (2021)
- [A Deep Value-network Based Approach for Multi-Driver Order Dispatching](https://mlsanity.com/paper/2106.04493/) (2021)
- [Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Health](https://mlsanity.com/paper/1912.13088/) (2019)
- [Variance-Aware Off-Policy Evaluation with Linear Function Approximation](https://mlsanity.com/paper/2106.11960/) (2021)
- [Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://mlsanity.com/paper/2202.10589/) (2022)
- [Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework](https://mlsanity.com/paper/2002.01711/) (2020)
- [Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2108.03706/) (2021)
- [Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learning](https://mlsanity.com/paper/2110.10719/) (2021)
- [Optimal policy evaluation using kernel-based temporal difference methods](https://mlsanity.com/paper/2109.12002/) (2021)
- [Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability](https://mlsanity.com/paper/2110.12343/) (2021)

## Abstract (arXiv)

This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra-cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q-function and policy estimators when the correlation structure is correctly specified, and (ii) their consistencies when the structure is mis-specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.