# Thompson Sampling-Based Learning and Control for Unknown Dynamic Systems

arXiv: https://arxiv.org/abs/2506.22186 · HTML: https://mlsanity.com/paper/2506.22186/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 24 of 24)

- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Predictive Entropy Search for Efficient Global Optimization of Black-box Functions](https://mlsanity.com/paper/1406.2541/) (2014)
- [Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control](https://mlsanity.com/paper/2111.10992/) (2021)
- [Fast active learning for pure exploration in reinforcement learning](https://mlsanity.com/paper/2007.13442/) (2020)
- [Formulas for Data-driven Control: Stabilization, Optimality and Robustness](https://mlsanity.com/paper/1903.06842/) (2019)
- [Scalable Thompson Sampling using Sparse Gaussian Process Models](https://mlsanity.com/paper/2006.05356/) (2020)
- [Taming Non-stationary Bandits: A Bayesian Approach](https://mlsanity.com/paper/1707.09727/) (2017)
- [Active Learning of Dynamics for Data-Driven Control Using Koopman Operators](https://mlsanity.com/paper/1906.05194/) (2019)
- [Learning Algorithms for Active Learning](https://mlsanity.com/paper/1708.00088/) (2017)
- [Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits](https://mlsanity.com/paper/1507.04523/) (2015)
- [On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RL](https://mlsanity.com/paper/2206.10770/) (2022)
- [Active Learning in Robotics: A Review of Control Principles](https://mlsanity.com/paper/2106.13697/) (2021)
- [Solving Bernoulli Rank-One Bandits with Unimodal Thompson Sampling](https://mlsanity.com/paper/1912.03074/) (2019)
- [Sliding-Window Thompson Sampling for Non-Stationary Settings](https://mlsanity.com/paper/2409.05181/) (2024)
- [Thompson Sampling for Linear-Quadratic Control Problems](https://mlsanity.com/paper/1703.08972/) (2017)
- [Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control](https://mlsanity.com/paper/2210.12583/) (2022)
- [Data-driven Methods Applied to Soft Robot Modeling and Control: A Review](https://mlsanity.com/paper/2305.12137/) (2023)
- [Localized active learning of Gaussian process state space models](https://mlsanity.com/paper/2005.02191/) (2020)
- [Thompson Sampling Achieves tilde O(√T) Regret in Linear Quadratic Control](https://mlsanity.com/paper/2206.08520/) (2022)
- [Robust exploration in linear quadratic reinforcement learning](https://mlsanity.com/paper/1906.01584/) (2019)
- [Direct data-driven control of LTV systems](https://mlsanity.com/paper/2111.02342/) (2021)
- [Sequential learning and control: Targeted exploration for robust performance](https://mlsanity.com/paper/2301.07995/) (2023)
- [Dual Control of Exploration and Exploitation for Auto-Optimisation Control with Active Learning](https://mlsanity.com/paper/2301.11984/) (2023)

## Abstract (arXiv)

Thompson sampling (TS) is a Bayesian randomized exploration strategy that samples options (e.g., system parameters or control laws) from the current posterior and then applies the selected option that is optimal for a task, thereby balancing exploration and exploitation; this makes TS effective for active learning-based controller design. However, TS relies on finite parametric representations, which limits its applicability to more general spaces, which are more commonly encountered in control system design. To address this issue, this work proposes a parameterization method for control law learning using reproducing kernel Hilbert spaces and designs a data-driven active learning control approach. Specifically, the proposed method treats the control law as an element in a function space, allowing the design of control laws without imposing restrictions on the system structure or the form of the controller. A TS framework is proposed in this work to reduce control costs through online exploration and exploitation, and the convergence guarantees are further provided for the learning process. Theoretical analysis shows that the proposed method learns the relationship between control laws and closed-loop performance metrics at an exponential rate, and the upper bound of control regret is also derived. Furthermore, the closed-loop stability of the proposed learning framework is analyzed. Numerical experiments on controlling unknown nonlinear systems validate the effectiveness of the proposed method.