mlsanityData to 7 Oct 2026

Learn Proportional Derivative Controllable Latent Space from Pixels

arXiv 2110.08239 ↗First posted 15 Oct 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in latent space dynamics model from pixels show promising progress in vision-based model predictive control (MPC). However, executing MPC in real time can be challenging due to its intensive computational cost in each timestep. We propose to introduce additional learning objectives to enforce that the learned latent space is proportional derivative controllable. In execution time, the simple PD-controller can be applied directly to the latent space encoded from pixels, to produce simple and effective control to systems with visual observations. We show that our method outperforms baseline methods to produce robust goal reaching and trajectory tracking in various environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

32 of 45 references.

References · 32

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2016OpenAI Gymcited by 1,684
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018World Modelscited by 1,444
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2018DeepMind Control Suitecited by 935
  9. 2016Tutorial on Variational Autoencoderscited by 404
  10. 2019Learning Latent Plans from Playcited by 310
  11. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  12. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
20 more
  1. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  2. 2020Neural Lyapunov Controlcited by 142
  3. 2016Variational Autoencoder for Deep Learning of Images, Labels and Captionscited by 137
  4. 2019Hamiltonian Generative Networkscited by 136
  5. 2019Learning Compositional Koopman Operators for Model-Based Controlcited by 67
  6. 2019Deep Visual MPC-Policy Learning for Navigationcited by 54
  7. 2020Learning Stability Certificates from Datacited by 45
  8. 2019Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Videocited by 43
  9. 2020Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Controlcited by 38
  10. 2017Robust Locally-Linear Controllable Embeddingcited by 37
  11. 2021Stabilizing Neural Control Using Self-Learned Almost Lyapunov Criticscited by 33
  12. 2018Probabilistic Video Generation using Holistic Attribute Controlcited by 24
  13. 2021Multimodal VAE Active Inference Controllercited by 21
  14. 2019Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Controlcited by 17
  15. 2020Predictive Coding for Locally-Linear Controlcited by 17
  16. 2017PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representationscited by 16
  17. 2020NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spacescited by 15
  18. 2017Model-Based Control Using Koopman Operatorscited by 14
  19. 2020q-VAE for Disentangled Representation Learning and Latent Dynamical Systemscited by 9
  20. 2020Neural Identification for Controlcited by 3

Built on by

Not cited yet.