mlsanityData to 14 Oct 2026

Prompt Optimization with Logged Bandit Data

arXiv 2504.02646 ↗First posted 3 Apr 2025cs.LG cs.AI cs.IRcited by 1

Not on the map. Why?

Abstract (arXiv)

We study how to use naturally available user feedback, such as clicks, to optimize large language model (LLM) pipelines for generating personalized sentences using prompts. Naive approaches, which estimate the policy gradient in the prompt space, suffer either from variance caused by the large action space of prompts or bias caused by inaccurate reward predictions. To circumvent these challenges, we propose a novel kernel-based off-policy gradient method, which estimates the policy gradient by leveraging similarity among generated sentences, substantially reducing variance while suppressing the bias. Empirical results on our newly established suite of benchmarks demonstrate the effectiveness of the proposed approach in generating personalized descriptions for movie recommendations, particularly when the number of candidate prompts is large.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

45 of 58 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2020Learning to summarize from human feedbackcited by 2,588
33 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2017Neural Collaborative Filteringcited by 1,111
  4. 2014Distributed Representations of Sentences and Documentscited by 678
  5. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  6. 2017On Kernelized Multi-armed Banditscited by 352
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  8. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 242
  9. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  10. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  11. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  12. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 137
  13. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  14. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  15. 2018Offline A/B testing for Recommender Systemscited by 96
  16. 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
  17. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  18. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  19. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
  20. 2020Off-policy Bandits with Deficient Supportcited by 61
  21. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 58
  22. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  23. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  24. 2024Efficient Exploration for LLMscited by 45
  25. 2024Prompt Optimization with Human Feedbackcited by 35
  26. 2022CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learningcited by 34
  27. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  28. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  29. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  30. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  31. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  32. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  33. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation
  2. 2024Personalized Language Modeling from Personalized Human Feedback
  3. 2025T-POP: Test-Time Personalization with Online Preference Feedback
  4. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalization
  5. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation
  6. 2025Language Model Personalization via Reward Factorization
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Search
  8. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models