# Finite Memory Belief Approximation for Optimal Control in Partially Observable Markov Decision Processes

arXiv: https://arxiv.org/abs/2601.03132 · HTML: https://mlsanity.com/paper/2601.03132/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 5 of 5)

- [Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2010.07452/) (2020)
- [Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces](https://mlsanity.com/paper/1503.02244/) (2015)
- [Learning-based Trajectory Tracking for Bird-inspired Flapping-Wing Robots](https://mlsanity.com/paper/2411.15130/) (2024)
- [RoVerFly: Robust and Versatile Implicit Hybrid Control of Quadrotor-Payload Systems](https://mlsanity.com/paper/2509.11149/) (2025)
- [Partially Observed Optimal Stochastic Control: Regularity, Optimality, Approximations, and Learning](https://mlsanity.com/paper/2412.06735/) (2024)

## Built on by (most cited first, 4 of 4)

- [WOMBET: World Model-Based Experience Transfer for Robust and Sample-efficient Reinforcement Learning](https://mlsanity.com/paper/2604.08958/) (2026)
- [Robust Adversarial Policy Optimization Under Dynamics Uncertainty](https://mlsanity.com/paper/2604.10974/) (2026)
- [Trajectory-Regularized Stochastic Optimal Control via KL Divergence](https://mlsanity.com/paper/2607.22201/) (2026)
- [In-Distribution Imagination for Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2609.38673/) (2026)

## Abstract (arXiv)

We study finite memory belief approximation for partially observable (PO) stochastic optimal control (SOC) problems. While belief states are sufficient for SOC in partially observable Markov decision processes (POMDPs), they are generally infinite-dimensional and impractical. We interpret truncated input-output (IO) histories as inducing a belief approximation and develop a metric-based theory that directly relates information loss to control performance. Using the Wasserstein metric, we derive policy-conditional performance bounds that quantify value degradation induced by finite memory along typical closed-loop trajectories. Our analysis proceeds via a fixed-policy comparison: we evaluate two cost functionals under the same closed-loop execution and isolate the effect of replacing the true belief by its finite memory approximation inside the belief-level cost. For linear quadratic Gaussian (LQG) systems, we provide closed-form belief mismatch evaluation and empirically validate the predicted mechanism, demonstrating that belief mismatch decays approximately exponentially with memory length and that the induced performance mismatch scales accordingly. Together, these results provide a metric-aware characterization of what finite memory belief approximation can and cannot achieve in PO settings.