# Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target

arXiv: https://arxiv.org/abs/2605.18899 · HTML: https://mlsanity.com/paper/2605.18899/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 11 of 11)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization](https://mlsanity.com/paper/2601.05242/) (2026)
- [Data-efficient Fine-tuning for LLM-based Recommendation](https://mlsanity.com/paper/2401.17197/) (2024)
- [On Softmax Direct Preference Optimization for Recommendation](https://mlsanity.com/paper/2406.09215/) (2024)
- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [Leveraging LLM Reasoning Enhances Personalized Recommender Systems](https://mlsanity.com/paper/2408.00802/) (2024)
- [CTR-Guided Generative Query Suggestion in Conversational Search](https://mlsanity.com/paper/2507.04072/) (2025)
- [From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System](https://mlsanity.com/paper/2508.15811/) (2025)
- [Reinforcement Learning from User Feedback](https://mlsanity.com/paper/2505.14946/) (2025)
- [Towards Trustworthy LLM-Based Recommendation via Rationale Integration](https://mlsanity.com/paper/2601.02364/) (2025)

## Built on by (most cited first, 1 of 1)

- [Causal Memory Policy: Making Memory Utility Identifiable by Intervening on Retrieval](https://mlsanity.com/paper/2610.02070/) (2026)

## Abstract (arXiv)

Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines.