Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization

Despite the sustained popularity of Q-learning as a practical tool for policy determination, a majority of relevant theoretical literature deals with either constant (ηtequiv η) or polynomially decaying (ηt = η t-α) learning schedules. However, it is well known that these choices suffer from either persistent bias or prohibitively slow convergence. In contrast, the recently proposed linear decay to zero (textttLD2Z: ηt,n=η(1-t/n)) schedule has shown appreciable empirical performance, but its theoretical and statistical properties remain largely unexplored, especially in the Q-learning setting. We address this gap in the literature by first considering a general class of power-law decay to zero (textttPD2Z-nu: ηt,n=η(1-t/n)nu). Proceeding step-by-step, we present a sharp non-asymptotic error bound for Q-learning with textttPD2Z-nu schedule, which then is used to derive a central limit theory for a new textittail Polyak-Ruppert averaging estimator. Finally, we also provide a novel time-uniform Gaussian approximation (also known as textitstrong invariance principle) for the partial sum process of Q-learning iterates, which facilitates bootstrap-based inference. All our theoretical results are complemented by extensive numerical experiments. Beyond being new theoretical and statistical contributions to the Q-learning literature, our results definitively establish that textttLD2Z and in general textttPD2Z-nu achieve a best-of-both-worlds property: they inherit the rapid decay from initialization (characteristic of constant step-sizes) while retaining the asymptotic convergence guarantees (characteristic of polynomially decaying schedules). This dual advantage explains the empirical success of textttLD2Z while providing practical guidelines for inference through our results.