mlsanityData to 7 Oct 2026

Learning What To Do by Simulating the Past

arXiv 2104.03946 ↗First posted 8 Apr 2021cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state. Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale it to complex tasks? In this work, we show that by combining a learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce a specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 8 Apr 2021.

Builds on

27 of 42 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
15 more
  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  4. 2017Population Based Training of Neural Networkscited by 351
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  6. 2018Generative Adversarial Imitation from Observationcited by 183
  7. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  8. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  9. 2018Probabilistic Recurrent State-Space Modelscited by 64
  10. 2018Learning Plannable Representations with Causal InfoGANcited by 56
  11. 2016Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Modelcited by 53
  12. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  13. 2019Preferences Implicit in the State of the Worldcited by 33
  14. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  15. 2019Learning from Observations Using a Single Video Demonstration and Human Feedbackcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 2.

4 papers.

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  2. 2023Inferring the Future by Imagining the Pastcited by 4
  3. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
  4. 2022Modeling Human Behavior Part I -- Learning and Belief Approachescited by 1