# DeepKoCo: Efficient latent planning with a task-relevant Koopman representation

arXiv: https://arxiv.org/abs/2011.12690 · HTML: https://mlsanity.com/paper/2011.12690/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Nov 2020.

## Built on by (most cited first, 5 of 5)

- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Task-Oriented Koopman-Based Control with Contrastive Encoder](https://mlsanity.com/paper/2309.16077/) (2023)
- [ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videos](https://mlsanity.com/paper/2403.08591/) (2024)
- [Learning Predictive Control with Deep Koopman Operators for Autonomous Vehicle Motion Planning](https://mlsanity.com/paper/2606.08136/) (2026)
- [Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality](https://mlsanity.com/paper/2609.32322/) (2026)

## Abstract (arXiv)

This paper presents DeepKoCo, a novel model-based agent that learns a latent Koopman representation from images. This representation allows DeepKoCo to plan efficiently using linear control methods, such as linear model predictive control. Compared to traditional agents, DeepKoCo learns task-relevant dynamics, thanks to the use of a tailored lossy autoencoder network that allows DeepKoCo to learn latent dynamics that reconstruct and predict only observed costs, rather than all observed dynamics. As our results show, DeepKoCo achieves similar final performance as traditional model-free methods on complex control tasks while being considerably more robust to distractor dynamics, making the proposed agent more amenable for real-life applications.