mlsanityData to 7 Oct 2026

Contextual Markov Decision Processes

arXiv 1502.02259 ↗First posted 8 Feb 2015stat.ML cs.LGcited by 213

Not on the map. Why?

Abstract (arXiv)

We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a customer's behavior when interacting with a website (the learner). The customer's behavior depends on gender, age, location, device, etc. Based on that behavior, the website objective is to determine customer characteristics, and to optimize the interaction between them. Our work focuses on one basic scenario--finite horizon with a small known number of possible contexts. We suggest a family of algorithms with provable guarantees that learn the underlying models and the latent contexts, and optimize the CMDPs. Bounds are obtained for specific naive implementations, and extensions of the framework are discussed, laying the ground for future research.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 19; 2022: 36; 2023: 41; 2024: 38; 2025: 43; 2026: 35.

213 papers.

Sort:

By area:

  1. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  2. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  3. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  4. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  5. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  6. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environmentcited by 60
  7. 2022Exploration via Elliptical Episodic Bonusescited by 51
  8. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  9. 2021Learning a Large Neighborhood Search Algorithm for Mixed Integer Programscited by 41
  10. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  11. 2022Efficient Risk-Averse Reinforcement Learningcited by 41
  12. 2022Automated Dynamic Algorithm Configurationcited by 37
38 more
  1. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  2. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  3. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  4. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 35
  5. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  6. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  7. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  8. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  9. 2022Factored Adaptation for Non-Stationary Reinforcement Learningcited by 30
  10. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  11. 2021DACBench: A Benchmark Library for Dynamic Algorithm Configurationcited by 28
  12. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  13. 2021TempoRL: Learning When to Actcited by 25
  14. 2021Block Contextual MDPs for Continual Learningcited by 25
  15. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  16. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  17. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  18. 2023Causal Reinforcement Learning: A Surveycited by 22
  19. 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
  20. 2023Universal Morphology Control via Contextual Modulationcited by 21
  21. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  22. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  23. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 19
  24. 2022Generalization in Cooperative Multi-Agent Systemscited by 18
  25. 2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
  26. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  27. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  28. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 16
  29. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  30. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  31. 2024LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Rewardcited by 16
  32. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 16
  33. 2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
  34. 2021Reinforcement Learning in Reward-Mixing MDPscited by 15
  35. 2022Learning Mixtures of Linear Dynamical Systemscited by 15
  36. 2022An Adaptive Deep RL Method for Non-Stationary Environments with Piecewise Stable Contextcited by 15
  37. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
  38. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14

50 most cited of 213 All 213 (JSON)