# Heteroscedastic Uncertainty for Robust Generative Latent Dynamics

arXiv: https://arxiv.org/abs/2008.08157 · HTML: https://mlsanity.com/paper/2008.08157/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor](https://mlsanity.com/paper/2311.01248/) (2023)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Randomized-to-Canonical Model Predictive Control for Real-world Visual Robotic Manipulation](https://mlsanity.com/paper/2207.01840/) (2022)
- [Working Backwards: Learning to Place by Picking](https://mlsanity.com/paper/2312.02352/) (2023)

## Abstract (arXiv)

Learning or identifying dynamics from a sequence of high-dimensional observations is a difficult challenge in many domains, including reinforcement learning and control. The problem has recently been studied from a generative perspective through latent dynamics: high-dimensional observations are embedded into a lower-dimensional space in which the dynamics can be learned. Despite some successes, latent dynamics models have not yet been applied to real-world robotic systems where learned representations must be robust to a variety of perceptual confounds and noise sources not seen during training. In this paper, we present a method to jointly learn a latent state representation and the associated dynamics that is amenable for long-term planning and closed-loop control under perceptually difficult conditions. As our main contribution, we describe how our representation is able to capture a notion of heteroscedastic or input-specific uncertainty at test time by detecting novel or out-of-distribution (OOD) inputs. We present results from prediction and control experiments on two image-based tasks: a simulated pendulum balancing task and a real-world robotic manipulator reaching task. We demonstrate that our model produces significantly more accurate predictions and exhibits improved control performance, compared to a model that assumes homoscedastic uncertainty only, in the presence of varying degrees of input degradation.