mlsanityData to 14 Oct 2026

Token-level Proximal Policy Optimization for Query Generation

arXiv 2411.00722 ↗First posted 1 Nov 2024cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

also in 1 more
Abstract (arXiv)

Query generation is a critical task for web search engines (e.g. Google, Bing) and recommendation systems. Recently, state-of-the-art query generation methods leverage Large Language Models (LLMs) for their strong capabilities in context understanding and text generation. However, they still face challenges in generating high-quality queries in terms of inferring user intent based on their web search interaction history. In this paper, we propose Token-level Proximal Policy Optimization (TPPO), a noval approach designed to empower LLMs perform better in query generation through fine-tuning. TPPO is based on the Reinforcement Learning from AI Feedback (RLAIF) paradigm, consisting of a token-level reward model and a token-level proximal policy optimization module to address the sparse reward challenge in traditional RLAIF frameworks. To evaluate the effectiveness and robustness of TPPO, we conducted experiments on both open-source dataset and an industrial dataset that was collected from a globally-used search engine. The experimental results demonstrate that TPPO significantly improves the performance of query generation for LLMs and outperforms its existing competitors.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 50 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

35 of 50 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
23 more
  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  2. 2023A Survey on Large Language Models for Recommendationcited by 379
  3. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 337
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  6. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  7. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  8. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  9. 2023LLMRec: Large Language Models with Graph Augmentation for Recommendationcited by 172
  10. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
  11. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  12. 2023GPT4Rec: A Generative Framework for Personalized Recommendation and User Interests Interpretationcited by 103
  13. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  14. 2023PALR: Personalization Aware LLMs for Recommendationcited by 96
  15. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  16. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 53
  17. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  18. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  19. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  20. 2024Can Small Language Models be Good Reasoners for Sequential Recommendation?cited by 26
  21. 2024Aligning Large Language Models via Fine-grained Supervisioncited by 13
  22. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Datacited by 12
  23. 2022Reproducing Personalised Session Search over the AOL Query Logcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  2. 2026PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimizationcited by 5
  3. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  4. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  5. 2026It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learningcited by 2
  6. 2026Supervised Fine-Tuning Needs to Unlock the Potential of Token Prioritycited by 0
  7. 2026Holder Policy Optimisationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking On-policy Optimization for Query Augmentation
  2. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs
  3. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searching
  4. 2024T-REG: Preference Optimization with Token-Level Reward Regularization
  5. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
  7. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
  8. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment