# Inverse Contextual Bandits without Rewards: Learning from a Non-Stationary Learner via Suffix Imitation

arXiv: https://arxiv.org/abs/2603.03778 · HTML: https://mlsanity.com/paper/2603.03778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Data Quality in Imitation Learning](https://mlsanity.com/paper/2306.02437/) (2023)
- [Beyond Reward: Offline Preference-guided Policy Optimization](https://mlsanity.com/paper/2305.16217/) (2023)
- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [A Contextual-bandit-based Approach for Informed Decision-making in Clinical Trials](https://mlsanity.com/paper/1809.00258/) (2018)
- [Listwise Reward Estimation for Offline Preference-based Reinforcement Learning](https://mlsanity.com/paper/2408.04190/) (2024)
- [Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward](https://mlsanity.com/paper/2206.06426/) (2022)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [How to Leverage Diverse Demonstrations in Offline Imitation Learning](https://mlsanity.com/paper/2405.17476/) (2024)
- [Neural Contextual Bandits for Personalized Recommendation](https://mlsanity.com/paper/2312.14037/) (2023)
- [SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning](https://mlsanity.com/paper/2402.13147/) (2024)
- [Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations](https://mlsanity.com/paper/2508.14383/) (2025)
- [IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution History](https://mlsanity.com/paper/2403.16075/) (2024)

## Abstract (arXiv)

We study the Inverse Contextual Bandit (ICB) problem, in which a learner seeks to optimize a policy while an observer, who cannot access the learner's rewards and only observes actions, aims to recover the underlying problem parameters. During the learning process, the learner's behavior naturally transitions from exploration to exploitation, resulting in non-stationary action data that poses significant challenges for the observer. To address this issue, we propose a simple and effective framework called Two-Phase Suffix Imitation. The framework discards data from an initial burn-in phase and performs empirical risk minimization using only data from a subsequent imitation phase. We derive a predictive decision loss bound that explicitly characterizes the bias-variance trade-off induced by the choice of burn-in length. Despite the severe information deficit, we show that a reward-free observer can achieve a convergence rate of tilde O(1/√N), matching the asymptotic efficiency of a fully reward-aware learner. This result demonstrates that a passive observer can effectively uncover the optimal policy from actions alone, attaining performance comparable to that of the learner itself.