# KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation

arXiv: https://arxiv.org/abs/2407.00548 · HTML: https://mlsanity.com/paper/2407.00548/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets](https://mlsanity.com/paper/2410.22325/) (2024)
- [Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy](https://mlsanity.com/paper/2511.00555/) (2025)
- [AsymDex: Asymmetry and Relative Coordinates for RL-based Bimanual Dexterity](https://mlsanity.com/paper/2411.13020/) (2024)
- [MetaKoopman: Bayesian Meta-Learning of Koopman Operators for Modeling Structured Dynamics under Distribution Shifts](https://mlsanity.com/paper/2607.26345/) (2026)

## Abstract (arXiv)

Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework. However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08× and the image-to-action Diffusion Policy by 1.16×. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: urlhttps://github.com/hychen-naza/KOROL.