# Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration

arXiv: https://arxiv.org/abs/2604.17457 · HTML: https://mlsanity.com/paper/2604.17457/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 40 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [The Value Function Polytope in Reinforcement Learning](https://mlsanity.com/paper/1901.11524/) (2019)
- [A First-Order Approach To Accelerated Value Iteration](https://mlsanity.com/paper/1905.09963/) (2019)
- [A Discrete-Time Switching System Analysis of Q-learning](https://mlsanity.com/paper/2102.08583/) (2021)
- [A Non-Asymptotic Theory of Seminorm Lyapunov Stability: From Deterministic to Stochastic Iterative Algorithms](https://mlsanity.com/paper/2502.14208/) (2025)
- [Final Iteration Convergence Bound of Q-Learning: Switching System Approach](https://mlsanity.com/paper/2205.05455/) (2022)
- [Dynamic Programming Through the Lens of Semismooth Newton-Type Methods (Extended Version)](https://mlsanity.com/paper/2203.08678/) (2022)
- [Finite-Time Analysis of Asynchronous Q-learning under Diminishing Step-Size from Control-Theoretic View](https://mlsanity.com/paper/2207.12217/) (2022)
- [Convex Programs and Lyapunov Functions for Reinforcement Learning: A Unified Perspective on the Analysis of Value-Based Methods](https://mlsanity.com/paper/2202.06922/) (2022)
- [Geometric Policy Iteration for Markov Decision Processes](https://mlsanity.com/paper/2206.05809/) (2022)
- [On Some Geometric Behavior of Value Iteration on the Orthant: Switching System Perspective](https://mlsanity.com/paper/2304.05597/) (2023)
- [On Value Iteration Convergence in Connected MDPs](https://mlsanity.com/paper/2406.09592/) (2024)
- [Analysis of Value Iteration Through Absolute Probability Sequences](https://mlsanity.com/paper/2502.03244/) (2025)
- [Geometric Re-Analysis of Classical MDP Solving Algorithms](https://mlsanity.com/paper/2503.04203/) (2025)

## Built on by (most cited first, 1 of 1)

- [Switching-Geometry Analysis of Deflated Q-Value Iteration](https://mlsanity.com/paper/2605.10811/) (2026)

## Abstract (arXiv)

Q-value iteration (Q-VI) is usually analyzed through the (γ)-contraction of the Bellman operator. This argument proves convergence to (Q^*), but it gives only a coarse account of when the induced greedy policy becomes optimal. We study discounted Q-VI as a switching system and focus on the practically optimal solution set (POSS), the set of (Q)-functions whose tie-broken greedy policies are optimal. The main result shows that Q-VI reaches the optimal action class in finite time by entering an invariant tube around (mathcal X1=Q^*+operatornamespan(mathbf 1)), which is contained in the POSS. For every (varepsilon>0), the distance to (mathcal X1) satisfies an exponential bound with rate ((barρ+varepsilon)k), where (barρ) is the joint spectral radius of the projected switching family restricted to directions transverse to (mathcal X1). When (barρ<γ), this transverse convergence is faster than the classical contraction rate. The analysis separates fast policy identification from the subsequent convergence to (Q^*), which may still be governed by the all-ones mode. We also give spectral and graph-theoretic conditions under which the strict inequality (barρ<γ) holds or fails.