# Solving infinite-horizon Dec-POMDPs using Finite State Controllers within JESP

arXiv: https://arxiv.org/abs/2109.08755 · HTML: https://mlsanity.com/paper/2109.08755/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 26 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 4 of 4)

- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [Solving POMDPs by Searching the Space of Finite Policies](https://mlsanity.com/paper/1301.6720/) (2013)
- [Solving POMDPs by Searching in Policy Space](https://mlsanity.com/paper/1301.7380/) (2013)
- [Policy Iteration for Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1401.3460/) (2014)

## Built on by (most cited first, 4 of 4)

- [Robust Robot Planning for Human-Robot Collaboration](https://mlsanity.com/paper/2302.13916/) (2023)
- [Monte-Carlo Search for an Equilibrium in Dec-POMDPs](https://mlsanity.com/paper/2305.11811/) (2023)
- [Decentralized Planning Using Probabilistic Hyperproperties](https://mlsanity.com/paper/2502.13621/) (2025)
- [Scalable Solution Methods for Dec-POMDPs with Deterministic Dynamics](https://mlsanity.com/paper/2508.21595/) (2025)

## Abstract (arXiv)

This paper looks at solving collaborative planning problems formalized as Decentralized POMDPs (Dec-POMDPs) by searching for Nash equilibria, i.e., situations where each agent's policy is a best response to the other agents' (fixed) policies. While the Joint Equilibrium-based Search for Policies (JESP) algorithm does this in the finite-horizon setting relying on policy trees, we propose here to adapt it to infinite-horizon Dec-POMDPs by using finite state controller (FSC) policy representations. In this article, we (1) explain how to turn a Dec-POMDP with N-1 fixed FSCs into an infinite-horizon POMDP whose solution is an Nth agent best response; (2) propose a JESP variant, called infJESP, using this to solve infinite-horizon Dec-POMDPs; (3) introduce heuristic initializations for JESP aiming at leading to good solutions; and (4) conduct experiments on state-of-the-art benchmark problems to evaluate our approach.