# AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents

arXiv: https://arxiv.org/abs/2504.09723 · HTML: https://mlsanity.com/paper/2504.09723/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 71 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 21 of 21)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning](https://mlsanity.com/paper/2404.18239/) (2024)
- [Offline A/B testing for Recommender Systems](https://mlsanity.com/paper/1801.07030/) (2018)
- [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025)
- [An Efficient Bandit Algorithm for Realtime Multivariate Optimization](https://mlsanity.com/paper/1810.09558/) (2018)
- [Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data](https://mlsanity.com/paper/2503.20749/) (2025)
- [WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents](https://mlsanity.com/paper/2404.05902/) (2024)
- [Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents](https://mlsanity.com/paper/2502.13012/) (2025)
- [An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors](https://mlsanity.com/paper/2309.14510/) (2023)
- [Inverse Reinforcement Learning from Summary Data](https://mlsanity.com/paper/1703.09700/) (2017)
- [Improving Website Hyperlink Structure Using Server Logs](https://mlsanity.com/paper/1512.07258/) (2015)
- [Understanding Collaborative Practices and Tools of Professional UX Practitioners in Software Organizations](https://mlsanity.com/paper/2302.11845/) (2023)

## Built on by (most cited first, 19 of 19)

- [OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation](https://mlsanity.com/paper/2506.05606/) (2025)
- [Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning](https://mlsanity.com/paper/2507.17842/) (2025)
- [Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping](https://mlsanity.com/paper/2510.07230/) (2025)
- [LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?](https://mlsanity.com/paper/2509.21501/) (2025)
- [MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces](https://mlsanity.com/paper/2510.08783/) (2025)
- [See, Think, Act: Online Shopper Behavior Simulation with VLM Agents](https://mlsanity.com/paper/2510.19245/) (2025)
- [ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents](https://mlsanity.com/paper/2605.16116/) (2026)
- [SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce](https://mlsanity.com/paper/2602.01443/) (2026)
- [Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding](https://mlsanity.com/paper/2505.05026/) (2025)
- [Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding](https://mlsanity.com/paper/2604.09581/) (2026)
- [SimPersona: Learning Discrete Buyer Personas from Raw Clickstreams for Grounded E-Commerce Agents](https://mlsanity.com/paper/2605.14205/) (2026)
- [SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents](https://mlsanity.com/paper/2605.19219/) (2026)
- [Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference](https://mlsanity.com/paper/2606.17165/) (2026)
- [OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation](https://mlsanity.com/paper/2510.03771/) (2025)
- [Improving Behavioral Alignment in LLM Social Simulations via Context Formation and Navigation](https://mlsanity.com/paper/2601.01546/) (2026)
- [Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems](https://mlsanity.com/paper/2604.07929/) (2026)
- [CHORUS: An Agentic Framework for Generating Realistic Deliberation Data](https://mlsanity.com/paper/2604.20651/) (2026)
- [Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation](https://mlsanity.com/paper/2608.02345/) (2026)
- [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026)

## Abstract (arXiv)

A/B testing experiment is a widely adopted method for evaluating UI/UX design decisions in modern web applications. Yet, traditional A/B testing remains constrained by its dependence on the large-scale and live traffic of human participants, and the long time of waiting for the testing result. Through formative interviews with six experienced industry practitioners, we identified critical bottlenecks in current A/B testing workflows. In response, we present AgentA/B, a novel system that leverages Large Language Model-based autonomous agents (LLM Agents) to automatically simulate user interaction behaviors with real webpages. AgentA/B enables scalable deployment of LLM agents with diverse personas, each capable of navigating the dynamic webpage and interactively executing multi-step interactions like search, clicking, filtering, and purchasing. In a demonstrative controlled experiment, we employ AgentA/B to simulate a between-subject A/B testing with 1,000 LLM agents Amazon.com, and compare agent behaviors with real human shopping behaviors at a scale. Our findings suggest AgentA/B can emulate human-like behavior patterns.