Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off
Not on the map. Why?
Abstract (arXiv)
A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 58 references that are in the map. First posted on arXiv 17 Dec 2022.
Builds on
28 of 58 references.
References · 28
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 211
- 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
- 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
- 2018Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulatorcited by 86
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
- 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
- 2019Making Deep Q-learning methods robust to time discretizationcited by 51
- 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
- 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
- 2018The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpointcited by 36
- 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 31
16 more
- 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 29
- 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 27
- 2020Control Frequency Adaptation via Action Persistence in Batch Reinforcement Learningcited by 27
- 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 21
- 2021Value Iteration in Continuous Actions, States and Timecited by 17
- 2019Adaptive Discretization for Episodic Reinforcement Learning in Metric Spacescited by 16
- 2019Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulatorcited by 15
- 2018Model-Free Linear Quadratic Control via Reduction to Expert Predictioncited by 10
- 2022Experimental Design for Linear Functionals in Reproducing Kernel Hilbert Spacescited by 9
- 2019Continuous-Time Markov Decision Processes with Controlled Observationscited by 7
- 2021Continuous-Time Fitted Value Iteration for Robust Policiescited by 6
- 2020Provably adaptive reinforcement learning in metric spacescited by 4
- 2022Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimalitycited by 4
- 2022Offline Reinforcement Learning at Multiple Frequenciescited by 4
- 2020Infinite-Horizon Linear-Quadratic-Gaussian Control with Costly Measurementscited by 3
- 2016Mean and variance of the LQG cost functioncited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.