# Learning Goals from Failure

arXiv: https://arxiv.org/abs/2006.15657 · HTML: https://mlsanity.com/paper/2006.15657/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 1 of 1)

- [Leveraging Self-Supervised Training for Unintentional Action Recognition](https://mlsanity.com/paper/2209.11870/) (2022)

## Abstract (arXiv)

We introduce a framework that predicts the goals behind observable human action in video. Motivated by evidence in developmental psychology, we leverage video of unintentional action to learn video representations of goals without direct supervision. Our approach models videos as contextual trajectories that represent both low-level motion and high-level action features. Experiments and visualizations show our trained model is able to predict the underlying goals in video of unintentional action. We also propose a method to "automatically correct" unintentional action by leveraging gradient signals of our model to adjust latent trajectories. Although the model is trained with minimal supervision, it is competitive with or outperforms baselines trained on large (supervised) datasets of successfully executed goals, showing that observing unintentional action is crucial to learning about goals in video. Project page: https://aha.cs.columbia.edu/