# Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking

arXiv: https://arxiv.org/abs/2603.22376 · HTML: https://mlsanity.com/paper/2603.22376/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 10 of 10)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [RankMixer: Scaling Up Ranking Models in Industrial Recommenders](https://mlsanity.com/paper/2507.15551/) (2025)
- [OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender](https://mlsanity.com/paper/2510.26104/) (2025)
- [Improving Training Stability for Multitask Ranking Models in Recommender Systems](https://mlsanity.com/paper/2302.09178/) (2023)
- [Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery](https://mlsanity.com/paper/2412.13667/) (2024)
- [Scaling Transformers for Discriminative Recommendation via Generative Pretraining](https://mlsanity.com/paper/2506.03699/) (2025)
- [Generative or Discriminative? Revisiting Text Classification in the Era of Transformers](https://mlsanity.com/paper/2506.12181/) (2025)
- [Climber: Toward Efficient Scaling Laws for Large Recommendation Models](https://mlsanity.com/paper/2502.09888/) (2025)

## Built on by (most cited first, 1 of 1)

- [Retrieve, Match, Escalate: Accurate and Scalable Product Linking with VLM-Distilled Cross-Encoders and Agentic VLMs](https://mlsanity.com/paper/2608.25037/) (2026)

## Abstract (arXiv)

We present an AI Co-Scientist framework that closes the research loop for the production search-ranking system of a large online travel platform -- pairing LLM agents with direct cloud-compute access so that idea generation, code implementation, GPU experimentation, and result analysis iterate end-to-end with a human scientist in the loop. The framework uses a hybrid agent architecture: single-LLM agents handle routine work, while multi-LLM consensus (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) is invoked for higher-stakes decisions. On the production ranking task, a human-designed transformer baseline (V2) yielded +0.118% over a pre-transformer baseline (V1); the AI Co-Scientist's automated loop on top of V2 contributed an additional +0.083%, for a combined +0.201% offline gain delivered in roughly one extra week of wall-clock time (single-run numbers; statistical limits discussed in the paper). The most useful AI proposals -- unified long-sequence layouts, slot-type embeddings, and multi-phase learning-rate schedules -- are standard practice in NLP and Vision but were absent from our production stack, suggesting that LLM agents can serve as cross-disciplinary connectors for ranking teams. We also report deployment context, negative results, and lessons learned.