# Repeated Principal-Agent Games with Unobserved Agent Rewards and Perfect-Knowledge Agents

arXiv: https://arxiv.org/abs/2304.07407 · HTML: https://mlsanity.com/paper/2304.07407/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 86 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 20 of 20)

- [Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems](https://mlsanity.com/paper/1405.2875/) (2014)
- [Inverse Optimization with Noisy Data](https://mlsanity.com/paper/1507.03266/) (2015)
- [Data-driven Inverse Optimization with Imperfect Information](https://mlsanity.com/paper/1512.05489/) (2015)
- [Learning to Bid Optimally and Efficiently in Adversarial First-price Auctions](https://mlsanity.com/paper/2007.04568/) (2020)
- [Data-Driven Estimation in Equilibrium Using Inverse Optimization](https://mlsanity.com/paper/1308.3397/) (2013)
- [An Online-Learning Approach to Inverse Optimization](https://mlsanity.com/paper/1810.12997/) (2018)
- [Multi-armed Bandit Problems with Strategic Arms](https://mlsanity.com/paper/1706.09060/) (2017)
- [Inverse Optimization: Closed-form Solutions, Geometry and Goodness of fit](https://mlsanity.com/paper/1511.04650/) (2015)
- [Generalized Inverse Optimization through Online Learning](https://mlsanity.com/paper/1810.01920/) (2018)
- [Exploration and Incentives in Reinforcement Learning](https://mlsanity.com/paper/2103.00360/) (2021)
- [Behavioral Analytics for Myopic Agents](https://mlsanity.com/paper/1702.05496/) (2017)
- [An Inverse Optimization Approach to Measuring Clinical Pathway Concordance](https://mlsanity.com/paper/1906.02636/) (2019)
- [Designing Truthful Contextual Multi-Armed Bandits based Sponsored Search Auctions](https://mlsanity.com/paper/2002.11349/) (2020)
- [An Optimal Bidimensional Multi-Armed Bandit Auction for Multi-unit Procurement](https://mlsanity.com/paper/1502.06934/) (2015)
- [Inducing Social Optimality in Games via Adaptive Incentive Design](https://mlsanity.com/paper/2204.05507/) (2022)
- [Optimal Auction Design for Flexible Consumers](https://mlsanity.com/paper/1607.02526/) (2016)
- [Socially-Optimal Mechanism Design for Incentivized Online Learning](https://mlsanity.com/paper/2112.14338/) (2021)
- [Statistics with Set-Valued Functions: Applications to Inverse Approximate Optimization](https://mlsanity.com/paper/1702.00708/) (2017)
- [Statistical Consistency of Set-Membership Estimator for Linear Systems](https://mlsanity.com/paper/1903.07552/) (2019)
- [Inverse Multiobjective Optimization Through Online Learning](https://mlsanity.com/paper/2010.06140/) (2020)

## Built on by (most cited first, 7 of 7)

- [Incentivized Learning in Principal-Agent Bandit Games](https://mlsanity.com/paper/2403.03811/) (2024)
- [Contractual Reinforcement Learning: Pulling Arms with Invisible Hands](https://mlsanity.com/paper/2407.01458/) (2024)
- [Learning to Mitigate Externalities: the Coase Theorem with Hindsight Rationality](https://mlsanity.com/paper/2406.19824/) (2024)
- [Principal-Agent Bandit Games with Self-Interested and Exploratory Learning Agents](https://mlsanity.com/paper/2412.16318/) (2024)
- [Learning to Incentivize in Repeated Principal-Agent Problems with Adversarial Agent Arrivals](https://mlsanity.com/paper/2505.23124/) (2025)
- [Fair Contracts in Principal-Agent Games with Heterogeneous Types](https://mlsanity.com/paper/2506.15887/) (2025)
- [Learning in Repeated Multi-Objective Stackelberg Games with Payoff Manipulation](https://mlsanity.com/paper/2508.14705/) (2025)

## Abstract (arXiv)

Motivated by a number of real-world applications from domains like healthcare and sustainable transportation, in this paper we study a scenario of repeated principal-agent games within a multi-armed bandit (MAB) framework, where: the principal gives a different incentive for each bandit arm, the agent picks a bandit arm to maximize its own expected reward plus incentive, and the principal observes which arm is chosen and receives a reward (different than that of the agent) for the chosen arm. Designing policies for the principal is challenging because the principal cannot directly observe the reward that the agent receives for their chosen actions, and so the principal cannot directly learn the expected reward using existing estimation techniques. As a result, the problem of designing policies for this scenario, as well as similar ones, remains mostly unexplored. In this paper, we construct a policy that achieves a low regret (i.e., square-root regret up to a log factor) in this scenario for the case where the agent has perfect-knowledge about its own expected rewards for each bandit arm. We design our policy by first constructing an estimator for the agent's expected reward for each bandit arm. Since our estimator uses as data the sequence of incentives offered and subsequently chosen arms, the principal's estimation can be regarded as an analogy of online inverse optimization in MAB's. Next we construct a policy that we prove achieves a low regret by deriving finite-sample concentration bounds for our estimator. We conclude with numerical simulations demonstrating the applicability of our policy to real-life setting from collaborative transportation planning.