mlsanityData to 7 Oct 2026

Reinforcement Learning with History-Dependent Dynamic Contexts

arXiv 2302.02061 ↗First posted 4 Feb 2023cs.LG cs.AI cs.SYcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce Dynamic Contextual Markov Decision Processes (DCMDPs), a novel reinforcement learning framework for history-dependent environments that generalizes the contextual MDP framework to handle non-Markov environments, where contexts change over time. We consider special cases of the model, with a focus on logistic DCMDPs, which break the exponential dependence on history length by leveraging aggregation functions to determine context transitions. This special structure allows us to derive an upper-confidence-bound style algorithm for which we establish regret bounds. Motivated by our theoretical results, we introduce a practical model-based algorithm for logistic DCMDPs that plans in a latent space and uses optimism over history-dependent features. We demonstrate the efficacy of our approach on a recommendation task (using MovieLens data) where user behavior dynamics evolve in response to recommendations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 25 of its 47 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

25 of 47 references.

References · 25

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  5. 2015Contextual Markov Decision Processescited by 213
  6. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  7. 2021Reinforcement learning based recommender systems: A surveycited by 187
  8. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  9. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  10. 2020Controllable Multi-Interest Framework for Recommendationcited by 92
  11. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  12. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
13 more
  1. 2020Instance-Wise Minimax-Optimal Algorithms for Logistic Banditscited by 56
  2. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  3. 2017Generalized Self-Concordant Functions: A Recipe for Newton-Type Methodscited by 33
  4. 2018Variance Reduction for Reinforcement Learning in Input-Driven Environmentscited by 30
  5. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 21
  6. 2021Sublinear Regret for Learning POMDPscited by 19
  7. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  8. 2021UCB-based Algorithms for Multinomial Logistic Regression Banditscited by 12
  9. 2021Confidence-Budget Matching for Sequential Budgeted Learningcited by 11
  10. 2022Reinforcement Learning in Presence of Discrete Markovian Context Evolutioncited by 9
  11. 2021Efficient PAC Reinforcement Learning in Regular Decision Processescited by 6
  12. 2019Advantage Amplification in Slowly Evolving Latent-State Environmentscited by 4
  13. 2022Reinforcement Learning with a Terminatorcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 5; 2026: 2.

11 papers.

  1. 2025Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learningcited by 6
  2. 2024DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learningcited by 3
  3. 2025Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysiscited by 2
  4. 2025Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPscited by 2
  5. 2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
  6. 2023Anytime-Competitive Reinforcement Learning with Policy Priorcited by 1
  7. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1
  8. 2023Ranking with Popularity Bias: User Welfare under Self-Amplification Dynamicscited by 0
  9. 2025Wavelet Predictive Representations for Non-Stationary Reinforcement Learningcited by 0
  10. 2025Reinforcement Learning and Consumption-Savings Behaviorcited by 0
  11. 2026Dynamic Context Scheduling: Learning Beyond the Static Universecited by 0