mlsanityData to 14 Oct 2026

Approximate information state based convergence analysis of recurrent Q-learning

arXiv 2306.05991 ↗First posted 9 Jun 2023cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In spite of the large literature on reinforcement learning (RL) algorithms for partially observable Markov decision processes (POMDPs), a complete theoretical understanding is still lacking. In a partially observable setting, the history of data available to the agent increases over time so most practical algorithms either truncate the history to a finite window or compress it using a recurrent neural network leading to an agent state that is non-Markovian. In this paper, it is shown that in spite of the lack of the Markov property, recurrent Q-learning (RQL) converges in the tabular setting. Moreover, it is shown that the quality of the converged limit depends on the quality of the representation which is quantified in terms of what is known as an approximate information state (AIS). Based on this characterization of the approximation error, a variant of RQL with AIS losses is presented. This variant performs better than a strong baseline for RQL that does not use AIS losses. It is demonstrated that there is a strong correlation between the performance of RQL over time and the loss associated with the AIS representation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 59 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

29 of 59 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2015Prioritized Experience Replaycited by 1,099
  9. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  10. 2018DeepMind Control Suitecited by 956
  11. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
17 more
  1. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  2. 2020Prioritized Level Replaycited by 198
  3. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  4. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 70
  5. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  6. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  7. 2020BeBold: Exploration Beyond the Boundary of Explored Regionscited by 33
  8. 2015Deep Attention Recurrent Q-Networkcited by 27
  9. 2021How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimationcited by 27
  10. 2020Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision Processescited by 23
  11. 2021Convergence of Finite Memory Q-Learning for POMDPs and Near Optimality of Learned Policies under Filter Stabilitycited by 22
  12. 2016Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networkscited by 21
  13. 2015Recurrent Reinforcement Learning: A Hybrid Approachcited by 13
  14. 2022Reinforcement Learning in Non-Markovian Environmentscited by 10
  15. 2016Learning to predict where to look in interactive environments using deep recurrent q-learningcited by 4
  16. 2018On the Complexity of Exploration in Goal-Driven Navigationcited by 4
  17. 2020Autonomous quadrotor obstacle avoidance based on dueling double deep recurrent Q-learning with monocular visioncited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  2. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  3. 2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
  4. 2023Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environmentscited by 7
  5. 2024Periodic agent-state based Q-learning for POMDPscited by 6