mlsanityData to 7 Oct 2026

Temporal Alignment for History Representation in Reinforcement Learning

arXiv 2204.03525 ↗First posted 7 Apr 2022cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Environments in Reinforcement Learning are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about the past. However, providing complete observations of numerous steps can be excessive. Inspired by human memory, we propose to represent history with only important changes in the environment and, in our approach, to obtain automatically this representation using self-supervision. Our method (TempAl) aligns temporally-close frames, revealing a general, slowly varying state of the environment. This procedure is based on contrastive loss, which pulls embeddings of nearby observations to each other while pushing away other samples from the batch. It can be interpreted as a metric that captures the temporal relations of observations. We propose to combine both common instantaneous and our history representation and we evaluate TempAl on all available Atari games from the Arcade Learning Environment. TempAl surpasses the instantaneous-only baseline in 35 environments out of 49. The source code of the method and of all the experiments is available at https://github.com/htdt/tempal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

25 of 35 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
13 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  6. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  7. 2020Whitening for Self-Supervised Representation Learningcited by 278
  8. 2021Mastering Atari Games with Limited Datacited by 235
  9. 2019Unsupervised State Representation Learning in Ataricited by 144
  10. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  11. 2020A unifying mutual information view of metric learning: cross-entropy vs. pairwise lossescited by 100
  12. 2014Unsupervised Learning of Spatiotemporally Coherent Metricscited by 45
  13. 2020Latent World Models For Intrinsically Motivated Explorationcited by 15

Built on by

Not cited yet.