mlsanityData to 14 Oct 2026

On learning history based policies for controlling Markov decision processes

arXiv 2211.03011 ↗First posted 6 Nov 2022cs.LG cs.SY eess.SYcited by 5

Not on the map. Why?

Abstract (arXiv)

Reinforcementlearning(RL)folkloresuggeststhathistory-basedfunctionapproximationmethods,suchas recurrent neural nets or history-based state abstraction, perform better than their memory-less counterparts, due to the fact that function approximation in Markov decision processes (MDP) can be viewed as inducing a Partially observable MDP. However, there has been little formal analysis of such history-based algorithms, as most existing frameworks focus exclusively on memory-less features. In this paper, we introduce a theoretical framework for studying the behaviour of RL algorithms that learn to control an MDP using history-based feature abstraction mappings. Furthermore, we use this framework to design a practical RL algorithm and we numerically evaluate its effectiveness on a set of continuous control tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 89 references that are in the map. First posted on arXiv 6 Nov 2022.

Builds on

29 of 89 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2018World Modelscited by 1,513
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
17 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2018Learning Dexterous In-Hand Manipulationcited by 687
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  5. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  6. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  7. 2017Neural Episodic Controlcited by 151
  8. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  9. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  10. 2021Reinforcement Learning, Bit by Bitcited by 59
  11. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  12. 2020Neuroevolution of Self-Interpretable Agentscited by 49
  13. 2015Deep Attention Recurrent Q-Networkcited by 27
  14. 2019Working Memory Graphscited by 23
  15. 2020Rapid Task-Solving in Novel Environmentscited by 23
  16. 2014Practical Kernel-Based Reinforcement Learningcited by 14
  17. 2014Extreme State Aggregation Beyond MDPscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 2.

5 papers.

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  2. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  3. 2023Approximate Information States for Worst-Case Control and Learning in Uncertain Systemscited by 3
  4. 2026Abstract Sim2Real through Approximate Information Statescited by 0
  5. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0