mlsanityData to 7 Oct 2026

Differentiating Bisimulation Metrics: A Framework for Parametric Markov Chain Fitting via Bicausal Optimal Transport

arXiv 2609.37239 ↗First posted 29 Sep 2026cs.LGcited by 0

On the map: World models → Latent predictive world models also fits: Latent dynamics models and model-based RL

Abstract (arXiv)

Many problems in sequential decision-making, such as imitation learning from observations, state-space compression, world-model learning, and sim-to-real transfer, can be reduced to learning a model such that a notion of distance with respect to the target process is minimized. We consider this general framework and consider the bisimulation metric, equivalently Bicausal Optimal Transport (BOT), as the notion of distance to minimize. We show that BOT, since it can be formulated as a linear program (LP), is differentiable with respect to the model dynamics. We then derive an exact closed-form gradient via the envelope theorem applied to the LP saddle point. The result is a general algorithm, Differentiable Bicausal Optimal Transport (D-BOT), that can be applied to each of the problems above. The proposed algorithm learns the best model by alternating between distance computation and gradient steps. We apply D-BOT for three different settings: state-space compression, parametric model learning, and imitation learning from observations (ILfO). We show empirical results that confirm the viability of all three instantiations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 14 of its 25 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

14 of 25 references.

References · 14

  1. 2018Behavioral Cloning from Observationcited by 482
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  3. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  4. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  5. 2020Primal Wasserstein Imitation Learningcited by 120
  6. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  7. 2023Optimal Transport for Offline Imitation Learningcited by 42
  8. 2022Approximate Policy Iteration with Bisimulation Metricscited by 11
  9. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  10. 2020Stochastic Optimization for Regularized Wasserstein Estimatorscited by 9
  11. 2023Imitation Learning from Observation through Optimal Transportcited by 7
  12. 2024Bisimulation Metrics are Optimal Transport Distances, and Can be Computed Efficientlycited by 6
2 more
  1. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  2. 2025Distances for Markov chains from sample streamscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimization
  2. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimally
  3. 2025Learning Causal States Under Partial Observability and Perturbation
  4. 2026Optimal Transport Meets Reinforcement Learning: A Survey
  5. 2025CCDP: Composition of Conditional Diffusion Policies with Guided Sampling
  6. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPs
  7. 2026Conditional Flow Matching for Transport Between Markov Processes
  8. 2025Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Training