# From Optimal Actions to World Models: Identifiability of Transition Kernels in Discounted MDPs

arXiv: https://arxiv.org/abs/2608.07301 · HTML: https://mlsanity.com/paper/2608.07301/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 2 of its 14 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 2 of 2)

- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [Inverting the Bellman Equation: From Q-Values to World Models](https://mlsanity.com/paper/2606.21173/) (2026)

## Abstract (arXiv)

We study what can be recovered about the transition probabilities of a Markov decision process from optimal actions alone. This is closely related to the inverse problem considered by Letcher et al., who ask when the dynamics can be recovered from numerical (Q)-values. Here the numerical values themselves are not observed; only the optimal actions are known, for every reward in a given class. For state-action rewards (r(s,a)), knowing the optimal actions for every reward also tells us how much better one action is than another when each is followed by the same fixed policy. This is still not enough to determine the transition probabilities uniquely. We prove that two kernels give the same optimal actions for every reward exactly when [ Qs,a = Bigl(Ps,a+tfrac1γ esmathsf T(L-I)Bigr)L-1 ] for one invertible matrix (L) satisfying (Lmathbf 1=mathbf 1). Near a kernel with strictly positive entries, there is an (n(n-1))-dimensional family of different kernels with this property. The result is unchanged if we consider only rewards having a unique optimal action at every state. We then compare this with rewards of the forms (r(s)) and (r(s,a,s')). Rewards that depend on the next state can usually recover the transition kernel itself: every row at a state with at least two actions is determined, and we describe exactly when a row at a state with one action can remain hidden. State rewards reveal less: two kernels give the same optimal actions exactly when every deterministic policy is optimal for the same set of rewards. The results show how the form of the reward affects what can be learned about the dynamics from optimal actions alone.