mlsanityData to 7 Oct 2026

Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL

arXiv 2407.01570 ↗First posted 27 May 2024cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite the significant advances in Deep Reinforcement Learning (RL) observed in the last decade, the amount of training experience necessary to learn effective policies remains one of the primary concerns in both simulated and real environments. Looking to solve this issue, previous work has shown that improved efficiency can be achieved by separately modeling the agent and environment, but usually requires a supervisory signal. In contrast to RL, humans can perfect a new skill from a small number of trials and often do so without a supervisory signal, making neuroscientific studies of human development a valuable source of inspiration for RL. In particular, we explore the idea of motor prediction, which states that humans develop an internal model of themselves and of the consequences that their motor commands have on the immediate sensory inputs. Our insight is that the movementofthe agent provides a cue that allows the duality between the agent and environment to be learned. To instantiate this idea, we present Ego-Foresight (EF), a self-supervised method for disentangling agent information based on motion and prediction. Our main finding is that, when used as an auxiliary task in feature learning, self-supervised agent awareness improves the sample-efficiency and performance of the underlying RL algorithm. To test our approach, we study the ability of EF to predict agent movement and disentangle agent information. Then, we integrate EF with model-free and model based RL algorithms to solve simulated control tasks, showing improved sample-efficiency and performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

37 of 60 references.

References · 37

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018DeepMind Control Suitecited by 935
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2017Deep Reinforcement Learning that Matterscited by 727
25 more
  1. 2018Disentangling by Factorisingcited by 614
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  4. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  5. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  6. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  7. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  8. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  9. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  10. 2022Masked World Models for Visual Controlcited by 169
  11. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 168
  12. 2017Unsupervised Learning of Disentangled Representations from Videocited by 138
  13. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  14. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 137
  15. 2022Disentangled Representation Learningcited by 111
  16. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  17. 2013Deep Learning of Representations: Looking Forwardcited by 84
  18. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  19. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  20. 2019End-to-End Pixel-Based Deep Active Inference for Body Perception and Actioncited by 24
  21. 2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
  22. 2020Robot self/other distinction: active inference meets neural networks learning in a mirrorcited by 12
  23. 2023Efficient RL via Disentangled Environment and Agent Representationscited by 11
  24. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
  25. 2019Morphology-Agnostic Visual Robotic Controlcited by 1

Built on by

Not cited yet.