# Operator Models for Continuous-Time Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.10383 · HTML: https://mlsanity.com/paper/2511.10383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 7 of its 98 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 7 of 7)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem](https://mlsanity.com/paper/2211.08875/) (2022)
- [Nonparametric approximation of conditional expectation operators](https://mlsanity.com/paper/2012.12917/) (2020)
- [Nonparametric Control Koopman Operators](https://mlsanity.com/paper/2405.07312/) (2024)
- [Towards Safe Robot Foundation Models Using Inductive Biases](https://mlsanity.com/paper/2505.10219/) (2025)

## Abstract (arXiv)

Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.