mlsanityData to 14 Oct 2026

Geometry of Drifting MDPs with Path-Integral Stability Certificates

arXiv 2601.21991 ↗First posted 29 Jan 2026cs.LG cs.AIcited by 8

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Real-world reinforcement learning is often nonstationary: rewards and dynamics drift, accelerate, oscillate, and trigger abrupt switches in the optimal action. Existing theory often represents nonstationarity with coarse-scale models that measure how much the environment changes, not how it changes locally -- even though acceleration and near-ties drive tracking error and policy chattering. We take a geometric view of nonstationary discounted Markov Decision Processes (MDPs) by modeling the environment as a differentiable homotopy path and tracking the induced motion of the optimal Bellman fixed point. This yields a length-curvature-kink signature of intrinsic complexity: cumulative drift, acceleration/oscillation, and action-gap-induced nonsmoothness. We prove a solver-agnostic path-integral stability bound and derive gap-safe feasible regions that certify local stability away from switch regimes. Building on these results, we introduce textitHomotopy-Tracking RL (HT-RL) and textitHT-MCTS, lightweight wrappers that estimate replay-based proxies of length, curvature, and near-tie proximity online and adapt learning or planning intensity accordingly. Experiments show improved tracking and dynamic regret over matched static baselines, with the largest gains in oscillatory and switch-prone regimes.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 29 Jan 2026.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamicscited by 5
  2. 2026Operator-Guided Invariance Learning for Continuous Reinforcement Learningcited by 5
  3. 2026Metric-Gradient Projection for Stable Multi-Agent Policy Learningcited by 4
  4. 2026Interactive Critique-Revision Training for Reliable Structured LLM Generationcited by 2
  5. 2026Matrix-Space Reinforcement Learning for Reusing Local Transition Geometrycited by 2
  6. 2026Learning Not to Optimize: Physics-Informed Action-Space Reshaping for Intent-Based Network Controlcited by 2
  7. 2026Differentiable Policy Transport over Multi-Layer Network Feasibility Geometrycited by 1
  8. 2026Geometry-Aware Operator Families for Structured Representation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Non-stationary and Varying-discounting Markov Decision Processes for Reinforcement Learning
  2. 2024On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
  3. 2026Local and Global Stability in Performative Reinforcement Learning
  4. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  5. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
  6. 2025Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis
  7. 2026Provable Offline Reinforcement Learning for Structured Cyclic MDPs
  8. 2024Solving robust MDPs as a sequence of static RL problems