mlsanityData to 7 Oct 2026

RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback

arXiv 2510.04096 ↗First posted 5 Oct 2025cs.IR cs.GT cs.LGcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

also in 1 more
Abstract (arXiv)

Competitive search is a setting where document publishers modify them to improve their ranking in response to a query. Recently, publishers have increasingly leveraged LLMs to generate and modify competitive content. We introduce Reinforcement Learning from Ranker Feedback (RLRF), a framework that trains LLMs using preference datasets derived from ranking competitions. The goal of a publisher (LLM-based) agent is to optimize content for improved ranking while accounting for the strategies of competing agents. We generate the datasets using approaches that do not rely on human-authored data. We show that our proposed agents consistently and substantially outperform previously suggested approaches for LLM-based competitive document modification. We further show that our agents are effective with ranking functions they were not trained for (i.e., out of distribution) and they adapt to strategic opponents. These findings provide support to the significant potential of using reinforcement learning in competitive search.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 57 of its 87 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

57 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025A Multi-Agent Perspective on Modern Information Retrieval20 Feb

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  9. 2024Qwen2.5 Technical Reportcited by 4,713
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
38 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  4. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  8. 2025Small Language Models are the Future of Agentic AIcited by 254
  9. 2023Large Language Model Alignment: A Surveycited by 241
  10. 2023Playing repeated games with Large Language Modelscited by 213
  11. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  12. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  13. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  14. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  15. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  16. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  17. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  18. 2025DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learningcited by 62
  19. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58
  20. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  21. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  22. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  23. 2023ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agentscited by 33
  24. 2023Applying BERT and ChatGPT for Sentiment Analysis of Lyme Disease in Scientific Literaturecited by 26
  25. 2024STEER: Assessing the Economic Rationality of Large Language Modelscited by 26
  26. 2024Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyondcited by 26
  27. 2025Overview of the TREC 2022 deep learning trackcited by 22
  28. 2024GLEE: A Unified Framework and Benchmark for Language-based Economic Environmentscited by 19
  29. 2024Economics Arena for Large Language Modelscited by 17
  30. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  31. 2025Guiding Retrieval using LLM-based Listwise Rankerscited by 16
  32. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learningcited by 13
  33. 2024Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Gamescited by 12
  34. 2024Embedding-Aligned Language Modelscited by 9
  35. 2019WIKIR: A Python toolkit for building a large-scale Wikipedia-based English Information Retrieval Datasetcited by 8
  36. 2023Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluationcited by 8
  37. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Modelscited by 7
  38. 2018Convergence of Learning Dynamics in Information Retrieval Gamescited by 5

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CSP: A Simulator For Multi-Agent Ranking Competitions
  2. 2026CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search
  3. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
  4. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
  5. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
  6. 2026CAPF: Guiding Search-Agent Rollouts with Credit-Attenuated Privileged Feedback
  7. 2026PaperScout: An Autonomous Agent for Academic Paper Search with Process-Aware Sequence-Level Policy Optimization
  8. 2026When should I search more: Adaptive Complex Query Optimization with Reinforcement Learning