mlsanityData to 7 Oct 2026

Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Information

arXiv 2212.12167 ↗First posted 23 Dec 2022stat.ML cs.LG stat.MEcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Motivated by the human-machine interaction such as training chatbots for improving customer satisfaction, we study human-guided human-machine interaction involving private information. We model this interaction as a two-player turn-based game, where one player (Alice, a human) guides the other player (Bob, a machine) towards a common goal. Specifically, we focus on offline reinforcement learning (RL) in this game, where the goal is to find a policy pair for Alice and Bob that maximizes their expected total rewards based on an offline dataset collected a priori. The offline setting presents two challenges: (i) We cannot collect Bob's private information, leading to a confounding bias when using standard RL methods, and (ii) a distributional mismatch between the behavior policy used to collect data and the desired policy we aim to learn. To tackle the confounding bias, we treat Bob's previous action as an instrumental variable for Alice's current decision making so as to adjust for the unmeasured confounding. We develop a novel identification result and use it to propose a new off-policy evaluation (OPE) method for evaluating policy pairs in this two-player turn-based game. To tackle the distributional mismatch, we leverage the idea of pessimism and use our OPE method to develop an off-policy learning algorithm for finding a desirable policy pair for both Alice and Bob. Finally, we prove that under mild assumptions such as partial coverage of the offline data, the policy pair obtained through our method converges to the optimal one at a satisfactory rate.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 51 references that are in the map. First posted on arXiv 23 Dec 2022.

Builds on

22 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes18 Sep
  2. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes26 May
  3. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning19 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 16

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  2. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  3. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  4. 2016Learning in games with continuous action sets and unknown payoff functionscited by 78
  5. 2018Confounding-Robust Policy Improvementcited by 73
  6. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  7. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  8. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  9. 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
  10. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  11. 2021Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?cited by 24
  12. 2016Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variablescited by 23
4 more
  1. 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 19
  2. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  3. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  4. 2017The GENIUS Approach to Robust Mendelian Randomization Inferencecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Preference-Based Apprenticeship Learning
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  4. 2024Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning
  5. 2023Learning to Influence Human Behavior with Offline Reinforcement Learning
  6. 2024Preserving Expert-Level Privacy in Offline Reinforcement Learning
  7. 2022User-Interactive Offline Reinforcement Learning
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog