mlsanityData to 14 Oct 2026

MultiScale Contextual Bandits for Long Term Objectives

arXiv 2503.17674 ↗First posted 22 Mar 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The feedback that AI systems (e.g., recommender systems, chatbots) collect from user interactions is a crucial source of training data. While short-term feedback (e.g., clicks, engagement) is widely used for training, there is ample evidence that optimizing short-term feedback does not necessarily achieve the desired long-term objectives. Unfortunately, directly optimizing for long-term objectives is challenging, and we identify the disconnect in the timescales of short-term interventions (e.g., rankings) and the long-term feedback (e.g., user retention) as one of the key obstacles. To overcome this disconnect, we introduce the framework of MultiScale Policy Learning to contextually reconcile that AI systems need to act and optimize feedback at multiple interdependent timescales. Following a PAC-Bayes motivation, we show how the lower timescales with more plentiful data can provide a data-dependent hierarchical prior for faster learning at higher scales, where data is more scarce. As a result, the policies at all levels effectively optimize for the long-term. We instantiate the framework with MultiScale Off-Policy Bandit Learning (MSBL) and demonstrate its effectiveness on three tasks relating to recommender and conversational systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 22 Mar 2025.

Builds on

31 of 52 references.

References · 31

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  3. 2016The Option-Critic Architecturecited by 448
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  6. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  7. 2019A Survey on Session-based Recommender Systemscited by 97
  8. 2020Feedback Loop and Bias Amplification in Recommender Systemscited by 90
  9. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  10. 2022KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videoscited by 72
  11. 2020Controlling Fairness and Bias in Dynamic Learning-to-Rankcited by 68
  12. 2016Cascading Bandits for Large-Scale Recommendation Problemscited by 45
19 more
  1. 2023KuaiSim: A Comprehensive Simulator for Recommender Systemscited by 30
  2. 2021What are you optimizing for? Aligning Recommender Systems with Human Valuescited by 29
  3. 2018Bayesian Counterfactual Risk Minimizationcited by 28
  4. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  5. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  6. 2020From Optimizing Engagement to Measuring Valuecited by 21
  7. 2023Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delaycited by 17
  8. 2023Reinforcing User Retention in a Billion Scale Short Video Recommender Systemcited by 15
  9. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  10. 2024Multi-Objective Recommendation via Multivariate Policy Learningcited by 10
  11. 2024Adaptive Decoding via Latent Preference Optimizationcited by 10
  12. 2021Top-k eXtreme Contextual Bandits with Arm Hierarchycited by 8
  13. 2023Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspectivecited by 8
  14. 2023Choosing the Right Weights: Balancing Value, Strategy, and Noise in Recommender Systemscited by 7
  15. 2022Nested banditscited by 5
  16. 2019Beyond Personalization: Research Directions in Multistakeholder Recommendationcited by 4
  17. 2024Pareto Front Approximation for Multi-Objective Session-Based Recommender Systemscited by 3
  18. 2023Ranking with Long-Term Constraintscited by 2
  19. 2024System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processescited by 2

Built on by

Not cited yet.