# Structuring Value Representations via Geometric Coherence in Markov Decision Processes

arXiv: https://arxiv.org/abs/2602.02978 · HTML: https://mlsanity.com/paper/2602.02978/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 3 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Built on by (most cited first, 6 of 6)

- [Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics](https://mlsanity.com/paper/2602.06939/) (2026)
- [Operator-Guided Invariance Learning for Continuous Reinforcement Learning](https://mlsanity.com/paper/2605.06500/) (2026)
- [Metric-Gradient Projection for Stable Multi-Agent Policy Learning](https://mlsanity.com/paper/2605.18809/) (2026)
- [Interactive Critique-Revision Training for Reliable Structured LLM Generation](https://mlsanity.com/paper/2605.08327/) (2026)
- [Matrix-Space Reinforcement Learning for Reusing Local Transition Geometry](https://mlsanity.com/paper/2605.14304/) (2026)
- [IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents](https://mlsanity.com/paper/2604.05157/) (2026)

## Abstract (arXiv)

Geometric properties can be leveraged to stabilize and speed reinforcement learning. Existing examples include encoding symmetry structure, geometry-aware data augmentation, and enforcing structural restrictions. In this paper, we take a novel view of RL through the lens of order theory and recast value function estimates into learning a desired poset (partially ordered set). We propose GCR-RL (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements -- by refining posets in previous steps and learning additional order relationships from temporal difference signals -- thus ensuring geometric coherence across the sequence of posets underpinning the learned value functions. Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements. Their theoretical properties and convergence rates are analyzed. We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.