# Stochastic Recursive Inclusions in two timescales with non-additive iterate dependent Markov noise

arXiv: https://arxiv.org/abs/1611.05961 · HTML: https://mlsanity.com/paper/1611.05961/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2016.

## Built on by (most cited first, 7 of 7)

- [Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications](https://mlsanity.com/paper/2401.09339/) (2024)
- [Online Learning with Adversaries: A Differential-Inclusion Analysis](https://mlsanity.com/paper/2304.01525/) (2023)
- [n-Step Temporal Difference Learning with Optimal n](https://mlsanity.com/paper/2303.07068/) (2023)
- [Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks](https://mlsanity.com/paper/2401.09665/) (2024)
- [Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning](https://mlsanity.com/paper/2110.12306/) (2021)
- [Reinforcement Learning with Quasi-Hyperbolic Discounting](https://mlsanity.com/paper/2409.10583/) (2024)
- [Adversary-Robust Learning from Fully Asynchronous Directional Derivative Estimates](https://mlsanity.com/paper/2605.09337/) (2026)

## Abstract (arXiv)

In this paper we study the asymptotic behavior of a stochastic approximation scheme on two timescales with set-valued drift functions and in the presence of non-additive iterate-dependent Markov noise. It is shown that the recursion on each timescale tracks the flow of a differential inclusion obtained by averaging the set-valued drift function in the recursion with respect to a set of measures which take into account both the averaging with respect to the stationary distributions of the Markov noise terms and the interdependence between the two recursions on different timescales. The framework studied in this paper builds on the works of itA. Ramaswamy et al. rm by allowing for the presence of non-additive iterate-dependent Markov noise. As an application, we consider the problem of computing the optimum in a constrained convex optimization problem where the objective function and the constraints are averaged with respect to the stationary distribution of an underlying Markov chain. Further the proposed scheme neither requires the differentiability of the objective function nor the knowledge of the averaging measure.