# On the Consistency of Kernel Methods with Dependent Observations

arXiv: https://arxiv.org/abs/2406.06101 · HTML: https://mlsanity.com/paper/2406.06101/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification](https://mlsanity.com/paper/1802.08334/) (2018)
- [Optimal Rates for Regularized Conditional Mean Embedding Learning](https://mlsanity.com/paper/2208.01711/) (2022)
- [Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon](https://mlsanity.com/paper/1812.11167/) (2018)
- [Metrizing Weak Convergence with Maximum Mean Discrepancies](https://mlsanity.com/paper/2006.09268/) (2020)
- [Learning with little mixing](https://mlsanity.com/paper/2206.08269/) (2022)
- [A Rigorous Theory of Conditional Mean Embeddings](https://mlsanity.com/paper/1912.00671/) (2019)
- [Actively Learning Gaussian Process Dynamics](https://mlsanity.com/paper/1911.09946/) (2019)
- [Learning Whenever Learning is Possible: Universal Learning under General Stochastic Processes](https://mlsanity.com/paper/1706.01418/) (2017)
- [Nonparametric approximation of conditional expectation operators](https://mlsanity.com/paper/2012.12917/) (2020)
- [On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions](https://mlsanity.com/paper/2205.13525/) (2022)
- [Regularised Least-Squares Regression with Infinite-Dimensional Output Space](https://mlsanity.com/paper/2010.10973/) (2020)
- [Probabilistic Robust Linear Quadratic Regulators with Gaussian Processes](https://mlsanity.com/paper/2105.07668/) (2021)

## Built on by (most cited first, 4 of 4)

- [Learning Ergodic Dynamical Systems from a Finite Trajectory](https://mlsanity.com/paper/2607.22399/) (2026)
- [Kernel conditional tests from learning-theoretic bounds](https://mlsanity.com/paper/2506.03898/) (2025)
- [Kernel-Based Learning of Safety Barriers](https://mlsanity.com/paper/2601.12002/) (2026)
- [Safety Certification is Classification](https://mlsanity.com/paper/2605.06087/) (2026)

## Abstract (arXiv)

The consistency of a learning method is usually established under the assumption that the observations are a realization of an independent and identically distributed (i.i.d.) or mixing process. Yet, kernel methods such as support vector machines (SVMs), Gaussian processes, or conditional kernel mean embeddings (CKMEs) all give excellent performance under sampling schemes that are obviously non-i.i.d., such as when data comes from a dynamical system. We propose the new notion of empirical weak convergence (EWC) as a general assumption explaining such phenomena for kernel methods. It assumes the existence of a random asymptotic data distribution and is a strict weakening of previous assumptions in the field. Our main results then establish consistency of SVMs, kernel mean embeddings, and general Hilbert-space valued empirical expectations with EWC data. Our analysis holds for both finite- and infinite-dimensional outputs, as we extend classical results of statistical learning to the latter case. In particular, it is also applicable to CKMEs. Overall, our results open new classes of processes to statistical learning and can serve as a foundation for a theory of learning beyond i.i.d. and mixing.