mlsanityData to 14 Oct 2026

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

arXiv 2609.39551 ↗First posted 30 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Auto-research agents, LLM systems that propose, implement, train, and evaluate model changes across iterations, promise to automate applied ML's experimental loop. Over long horizons, execution accuracy is a binding constraint: a change can silently leak held-out data, omit normalization, disconnect a gradient, or leave a train/eval flag unwired, invalidating expensive runs and compounding error across iterations. We present RankEvolve, an auto-research framework for evolving generative ranking models. An Executable Operating Protocol (EOP) declares phases, gates, branches, and loops, and the runtime enforces the compiled state machine. A meta-meta-harness composes complete black-box coding-agent products, including Claude Code and Codex, as execution-graph nodes that review and repair one another's work. In a budget-matched evaluation, heterogeneous composition raises all-oracle execution accuracy from the best single-product baseline of 45.8 percent to 62.5 percent (paired +16.7 points, 95 percent CI [6.6, 26.7]) while achieving a 10.4 percent silent critical-defect rate. An implemented knowledge layer carries findings, including negative results, across iterations. In a twelve-iteration deployment on the open-source HSTU recommender, RankEvolve reported NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48 percent over the published anchor) and 0.1948 on BASE (+2.80 percent). ExecML-HSTU, seeded by incidents from that deployment, provides the oracle benchmark for the execution-accuracy evaluation. A pre-specified LitGPT transfer split replicates the heterogeneous-composition effect beyond recommendation (+12.5 points, 95 percent CI [3.0, 22.0]), and a paired ablation isolates per-step from full-protocol instruction injection. These results characterize when runtime-controlled composition of coding-agent products improves execution accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 61 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

48 of 61 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench3 Jul
  2. 2025MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement27 May
  3. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research26 May
  4. 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search10 Apr
  5. 2024Agent Workflow Memory11 Sep
  6. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation5 Oct

Other references · 42

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
30 more
  1. 2018Self-Attentive Sequential Recommendationcited by 1,184
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  3. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  4. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  5. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  6. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 964
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  10. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  11. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  12. 2024AFlow: Automating Agentic Workflow Generationcited by 432
  13. 2024Automated Design of Agentic Systemscited by 407
  14. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
  15. 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
  16. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 226
  17. 2023Recommender Systems with Generative Retrievalcited by 198
  18. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
  19. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  20. 2019Multi-Interest Network with Dynamic Routing for Recommendation at Tmallcited by 114
  21. 2025Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solvingcited by 94
  22. 2019The Unfairness of Popularity Bias in Recommendationcited by 70
  23. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  24. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agentscited by 62
  25. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflowscited by 57
  26. 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 29
  27. 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 25
  28. 2026Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agentscited by 19
  29. 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedbackcited by 10
  30. 2026RankEvolve: Automating the Discovery of Retrieval Algorithms via LLM-Driven Evolutioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic ML Exploration (A-MLE) for Ads Ranking
  2. 2026Decoding ML Decision: An Agentic Reasoning Framework for Large-Scale Ranking System
  3. 2026AutoScientist-Quant: Self-Evolving Coding Agents for Automatic Research in Quantitative Investment
  4. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  5. 2026Scaling Automatic Research Agents via World Models
  6. 2026FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
  7. 2025ArchPilot: A Proxy-Guided Multi-Agent Approach for Machine Learning Engineering
  8. 2026Efficient Benchmarking of AI Agents