# Geometry of Drifting MDPs with Path-Integral Stability Certificates

arXiv: https://arxiv.org/abs/2601.21991 · HTML: https://mlsanity.com/paper/2601.21991/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 29 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Built on by (most cited first, 8 of 8)

- [Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics](https://mlsanity.com/paper/2602.06939/) (2026)
- [Operator-Guided Invariance Learning for Continuous Reinforcement Learning](https://mlsanity.com/paper/2605.06500/) (2026)
- [Metric-Gradient Projection for Stable Multi-Agent Policy Learning](https://mlsanity.com/paper/2605.18809/) (2026)
- [Interactive Critique-Revision Training for Reliable Structured LLM Generation](https://mlsanity.com/paper/2605.08327/) (2026)
- [Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry](https://mlsanity.com/paper/2605.14304/) (2026)
- [Learning Not to Optimize: Physics-Informed Action-Space Reshaping for Intent-Based Network Control](https://mlsanity.com/paper/2608.00908/) (2026)
- [Differentiable Policy Transport over Multi-Layer Network Feasibility Geometry](https://mlsanity.com/paper/2609.26068/) (2026)
- [Geometry-Aware Operator Families for Structured Representation Learning](https://mlsanity.com/paper/2609.33089/) (2026)

## Abstract (arXiv)

Real-world reinforcement learning is often nonstationary: rewards and dynamics drift, accelerate, oscillate, and trigger abrupt switches in the optimal action. Existing theory often represents nonstationarity with coarse-scale models that measure how much the environment changes, not how it changes locally -- even though acceleration and near-ties drive tracking error and policy chattering. We take a geometric view of nonstationary discounted Markov Decision Processes (MDPs) by modeling the environment as a differentiable homotopy path and tracking the induced motion of the optimal Bellman fixed point. This yields a length-curvature-kink signature of intrinsic complexity: cumulative drift, acceleration/oscillation, and action-gap-induced nonsmoothness. We prove a solver-agnostic path-integral stability bound and derive gap-safe feasible regions that certify local stability away from switch regimes. Building on these results, we introduce textitHomotopy-Tracking RL (HT-RL) and textitHT-MCTS, lightweight wrappers that estimate replay-based proxies of length, curvature, and near-tie proximity online and adapt learning or planning intensity accordingly. Experiments show improved tracking and dynamic regret over matched static baselines, with the largest gains in oscillatory and switch-prone regimes.