mlsanityData to 7 Oct 2026

GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents

arXiv 2511.00802 ↗First posted 2 Nov 2025cs.SE cs.CL cs.LGcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

With data-driven development now widely adopted, online A/B testing is an established method for measuring the effects of new technologies. However, deploying online experiments demands resources for design, implementation, and deployment, and may negatively impact users (e.g., unsafe or unethical outcomes) while requiring weeks of data collection. To address this, the growing research area of off-policy evaluation (OPE), or offline A/B testing, assesses new technologies offline using previously collected logged data. OPE is also a fundamental problem in reinforcement learning and is important where online testing is expensive or risky, such as healthcare, recommender systems, education, and robotics. Despite advances in code-generation large language models (LLMs) and agentic workflows, little is known about whether and how LLMs and LLM-based agents can automatically optimize OPE implementations. We propose GrowthHacker, a benchmark that evaluates baseline LLMs and LLM-based agents on large-scale public datasets. GrowthHacker autonomously and iteratively modifies code, runs OPE, and uses the metrics to guide subsequent optimization. We evaluate methods on Open Bandit Pipeline (OBP) and Scope-RL, and develop a two_agent framework that addresses limitations of existing frameworks while reducing complexity. Across both libraries, two_agent shows the highest reliability (98.1%-100% success rate) and positive-outcome rate (78%), with a median improvement of 4.4% among positive outcomes; CrewAI achieves the highest average improvement (37.9%) and is the only framework with zero extreme-value failures. AutoGen and Default each reach 65% positive-outcome rates. These results establish the feasibility of using LLM-based agents as automated "growth hackers" to continuously improve OPE systems, with implications for scaling data-driven decision-making where manual optimization is expensive.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 21 of its 52 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

21 of 52 references.

References · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  8. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  9. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  10. 2018Offline A/B testing for Recommender Systemscited by 94
  11. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  12. 2020Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluationcited by 77
9 more
  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  2. 2023RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repaircited by 43
  3. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
  4. 2024Search-Based LLMs for Code Optimizationcited by 27
  5. 2023RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shotcited by 14
  6. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  7. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  8. 2021Controlled Experimentation in Continuous Experimentation: Knowledge and Challengescited by 1
  9. 2023AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement Engineeringcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
  2. 2026OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents
  3. 2021Active Offline Policy Selection
  4. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  5. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  6. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2026HarnessOpt-Bench: Evaluating LLMs at Harness Optimization