mlsanityData to 7 Oct 2026

Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics

arXiv 2602.06939 ↗First posted 6 Feb 2026cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Non-Markovian dynamics are commonly found in real-world environments due to long-range dependencies, partial observability, and memory effects. The Bellman equation that is the central pillar of Reinforcement learning (RL) becomes only approximately valid under Non-Markovian. Existing work often focus on practical algorithm designs and offer limited theoretical treatment to address key questions, such as what dynamics are indeed capturable by the Bellman framework and how to inspire new algorithm classes with optimal approximations. In this paper, we present a novel topological viewpoint on temporal-difference (TD) based RL. We show that TD errors can be viewed as 1-cochain in the topological space of state transitions, while Markov dynamics are then interpreted as topological integrability. This novel view enables us to obtain a Hodge-type decomposition of TD errors into an integrable component and a topological residual, through a Bellman-de Rham projection. We further propose HodgeFlow Policy Search (HFPS) by fitting a potential network to minimize the non-integrable projection residual in RL, achieving stability/sensitivity guarantees. In numerical evaluations, HFPS is shown to significantly improve RL performance under non-Markovian.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

31 of 57 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
19 more
  1. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  2. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  3. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  4. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  5. 2020R-MADDPG for Partially Observable Environments and Limited Communicationcited by 36
  6. 2013Learning Finite-State Controllers for Partially Observable Environmentscited by 18
  7. 2024Modeling Other Players with Bayesian Beliefs for Games with Incomplete Informationcited by 16
  8. 2025Network Diffuser for Placing-Scheduling Service Function Chains with Inverse Demonstrationcited by 11
  9. 2025Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Taskscited by 8
  10. 2026Geometry of Drifting MDPs with Path-Integral Stability Certificatescited by 7
  11. 2024Cooperative Backdoor Attack in Decentralized Reinforcement Learning with Theoretical Guaranteecited by 6
  12. 2026Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learningcited by 6
  13. 2026Structuring Value Representations via Geometric Coherence in Markov Decision Processescited by 6
  14. 2025MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planningcited by 5
  15. 2025Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guaranteescited by 4
  16. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  17. 2024A policy iteration algorithm for non-Markovian control problemscited by 2
  18. 2025End-to-End Learning Framework for Solving Non-Markovian Optimal Controlcited by 2
  19. 2025Scaling Internal-State Policy-Gradient Methods for POMDPscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026NonZero: Interaction-Guided Exploration for Multi-Agent Monte Carlo Tree Searchcited by 4
  2. 2026Operator-Guided Invariance Learning for Continuous Reinforcement Learningcited by 4
  3. 2026Metric-Gradient Projection for Stable Multi-Agent Policy Learningcited by 3
  4. 2026Interactive Critique-Revision Training for Reliable Structured LLM Generationcited by 2
  5. 2026Matrix-Space Reinforcement Learning for Reusing Local Transition Geometrycited by 0