Data to 14 Oct 2026

Markov Decision Processes with Continuous Side Information

arXiv 1711.05726 ↗First posted 15 Nov 2017stat.ML cs.AI cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where baseline measurements of a patient at the start of a treatment episode form the context that may provide information about how the patient might respond to treatment decisions. We propose algorithms for learning in such Contextual Markov Decision Processes (CMDPs) under an assumption that the unobserved MDP parameters vary smoothly with the observed context. We also give lower and upper PAC bounds under the smoothness assumption. Because our lower bound has an exponential dependence on the dimension, we consider a tractable linear setting where the context is used to create linear combinations of a finite set of MDPs. For the linear setting, we give a PAC learning algorithm based on KWIK learning techniques.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 9; 2022: 10; 2023: 6; 2024: 7; 2025: 11; 2026: 12.

56 papers.

Sort:

By area:

2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
2022Evolving Curricula with Regret-Based Environment Designcited by 173
2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 46
2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 42
2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
2021TempoRL: Learning When to Actcited by 25
2021Block Contextual MDPs for Continual Learningcited by 25
2021Metadata-based Multi-Task Bandits with Bayesian Hierarchical Modelscited by 24
2021A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learningcited by 22
2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
2023Domain Randomization via Entropy Maximizationcited by 19

38 more

2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 17
2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 17
2024Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalizationcited by 16
2019Inverse Reinforcement Learning in Contextual MDPscited by 12
2022Optimism in Face of a Context: Regret Guarantees for Stochastic Contextual MDPcited by 11
2022Non-Linear Reinforcement Learning in Large Action Spaces: Structural Conditions and Sample-efficiency of Posterior Samplingcited by 10
2022Eluder-based Regret for Stochastic Contextual MDPscited by 9
2023Efficient Rate Optimal Regret for Adversarial Contextual MDPs Using Online Function Approximationcited by 9
2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9
2024Offline Oracle-Efficient Learning for Contextual MDPs via Layerwise Exploration-Exploitation Tradeoffcited by 8
2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 7
2024Model-Based Transfer Learning for Contextual Reinforcement Learningcited by 7
2024Sample Complexity Characterization for Linear Contextual MDPscited by 5
2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learningcited by 5
2025Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalizationcited by 5
2025Regret Bounds for Adversarial Contextual Bandits with General Function Approximation and Delayed Feedbackcited by 5
2022Multi-User Reinforcement Learning with Low Rank Rewardscited by 4
2022Hyperparameters in Contextual RL are Highly Situationalcited by 4
2025Can Context Bridge the Reality Gap? Sim-to-Real Transfer of Context-Aware Policiescited by 4
2022Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regretcited by 3
2022Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximationcited by 2
2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
2025When Is Diversity Rewarded in Cooperative Multi-Agent Learning?cited by 2
2025Learning to Decide with Just Enough: Information-Theoretic Context Summarization for CMDPscited by 2
2025Curriculum Design for Trajectory-Constrained Agent: Compressing Chain-of-Thought Tokens in LLMscited by 2
2026Structure Detection for Contextual Reinforcement Learningcited by 2
2026Near-Optimal Regret for Policy Optimization in Contextual MDPs with General Offline Function Approximationcited by 2
2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
2023Scaling Distributed Multi-task Reinforcement Learning with Experience Sharingcited by 1
2023JoinGym: An Efficient Query Optimization Environment for Reinforcement Learningcited by 1
2025On the Fly Adaptation of Behavior Tree-Based Policies through Reinforcement Learningcited by 1
2026Contextual Intelligence The Next Leap for Reinforcement Learningcited by 1
2026Contextual Multi-Task Reinforcement Learning for Autonomous Reef Monitoringcited by 1
2025Multi-Objective Planning with Contextual Lexicographic Reward Preferencescited by 0
2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
2026TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learningcited by 0
2026C-IDS: Solving Contextual POMDP via Information-Directed Objectivecited by 0

50 most cited of 56 All 56 (JSON)