# Learning Lifted Action Models from Unsupervised Visual Traces

arXiv: https://arxiv.org/abs/2604.19043 · HTML: https://mlsanity.com/paper/2604.19043/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary](https://mlsanity.com/paper/1705.00154/) (2017)
- [PDDLGym: Gym Environments from PDDL Problems](https://mlsanity.com/paper/2002.06432/) (2020)
- [Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning](https://mlsanity.com/paper/2105.14074/) (2021)
- [VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning](https://mlsanity.com/paper/2410.23156/) (2024)
- [Learning First-Order Symbolic Representations for Planning from the Structure of the State Space](https://mlsanity.com/paper/1909.05546/) (2019)
- [Learning First-Order Representations for Planning from Black-Box States: New Results](https://mlsanity.com/paper/2105.10830/) (2021)
- [Classical Planning in Deep Latent Space](https://mlsanity.com/paper/2107.00110/) (2021)
- [Safe Learning of Lifted Action Models](https://mlsanity.com/paper/2107.04169/) (2021)
- [Discovering User-Interpretable Capabilities of Black-Box Planning Agents](https://mlsanity.com/paper/2107.13668/) (2021)
- [From Real World to Logic and Back: Learning Generalizable Relational Concepts For Long Horizon Robot Planning](https://mlsanity.com/paper/2402.11871/) (2024)
- [Learning Lifted STRIPS Models from Action Traces Alone: A Simple, General, and Scalable Solution](https://mlsanity.com/paper/2411.14995/) (2024)
- [Learning Neural-Symbolic Descriptive Planning Models via Cube-Space Priors: The Voyage Home (to STRIPS)](https://mlsanity.com/paper/2004.12850/) (2020)
- [Towards Stable Symbol Grounding with Zero-Suppressed State AutoEncoder](https://mlsanity.com/paper/1903.11277/) (2019)

## Built on by (most cited first, 3 of 3)

- [Differentiable Learning of Lifted Action Schemas for Classical Planning](https://mlsanity.com/paper/2605.13282/) (2026)
- [Neurosymbolic Action Model Learning under Partial Observability](https://mlsanity.com/paper/2609.25766/) (2026)
- [Neurosymbolic Learning of Interpretable World Models](https://mlsanity.com/paper/2610.19955/) (2026)

## Abstract (arXiv)

Efficient construction of models capturing the preconditions and effects of actions is essential for applying AI planning in real-world domains. Extensive prior work has explored learning such models from high-level descriptions of state and/or action sequences. In this paper, we tackle a more challenging setting: learning lifted action models from sequences of state images, without action observation. We propose a deep learning framework that jointly learns state prediction, action prediction, and a lifted action model. We also introduce a mixed-integer linear program (MILP) to prevent prediction collapse and self-reinforcing errors among predictions. The MILP takes the predicted states, actions, and action model over a subset of traces and solves for logically consistent states, actions, and action model that are as close as possible to the original predictions. Pseudo-labels extracted from the MILP solution are then used to guide further training. Experiments across multiple domains show that integrating MILP-based correction helps the model escape local optima and converge toward globally consistent solutions.