# A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

arXiv: https://arxiv.org/abs/2509.25609 · HTML: https://mlsanity.com/paper/2509.25609/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 31)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions](https://mlsanity.com/paper/2308.11483/) (2023)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024)
- [Dissecting Adversarial Robustness of Multimodal LM Agents](https://mlsanity.com/paper/2406.12814/) (2024)
- [Attacking Vision-Language Computer Agents via Pop-ups](https://mlsanity.com/paper/2411.02391/) (2024)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [Adversarial Demonstration Attacks on Large Language Models](https://mlsanity.com/paper/2305.14950/) (2023)
- [SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models](https://mlsanity.com/paper/2305.19308/) (2023)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations](https://mlsanity.com/paper/2504.14150/) (2025)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)
- [How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?](https://mlsanity.com/paper/2402.07282/) (2024)
- [Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgments](https://mlsanity.com/paper/2406.00092/) (2024)
- [Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function](https://mlsanity.com/paper/2406.01382/) (2024)
- [When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1](https://mlsanity.com/paper/2410.01792/) (2024)
- [Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models](https://mlsanity.com/paper/2410.20745/) (2024)
- [PAARS: Persona Aligned Agentic Retail Shoppers](https://mlsanity.com/paper/2503.24228/) (2025)
- [LiLiuM: eBay's Large Language Models for e-commerce](https://mlsanity.com/paper/2406.12023/) (2024)
- [LLM Agents Are Hypersensitive to Nudges](https://mlsanity.com/paper/2505.11584/) (2025)

## Built on by (most cited first, 12 of 12)

- [How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors](https://mlsanity.com/paper/2601.21961/) (2026)
- [Visual Persuasion: What Influences Decisions of Vision-Language Models?](https://mlsanity.com/paper/2602.15278/) (2026)
- [Position: Behavioral Systems Require Behavioral Tests](https://mlsanity.com/paper/2608.18081/) (2026)
- [Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls](https://mlsanity.com/paper/2608.27727/) (2026)
- [Mecha-nudges for Machines](https://mlsanity.com/paper/2603.23433/) (2026)
- [Agent-Facing Information Design in LLM Tool Registries](https://mlsanity.com/paper/2605.23916/) (2026)
- [LLM Consumer Behavior Theory: Foundations of a Novel Research Field](https://mlsanity.com/paper/2606.18005/) (2026)
- [Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Science](https://mlsanity.com/paper/2608.10030/) (2026)
- [The Delegation Blind Spot: Auditing Product Decisions from Agent Choices](https://mlsanity.com/paper/2609.26642/) (2026)
- [CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments](https://mlsanity.com/paper/2609.27273/) (2026)
- [Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It](https://mlsanity.com/paper/2610.03195/) (2026)
- [Learning to Clarify Underspecified Intents Under Limited Interaction](https://mlsanity.com/paper/2610.04719/) (2026)

## Abstract (arXiv)

Environments built for people are increasingly operated by a new class of economic actors: LLM-powered software agents making decisions on our behalf. These decisions range from our purchases to travel plans to medical treatment selection. Current evaluations of these agents largely focus on task competence, but we argue for a deeper assessment: how these agents choose when faced with realistic decisions. We introduce ABxLab, a framework for systematically probing agentic choice through controlled manipulations of option attributes and persuasive cues. We apply this to a realistic web-based shopping environment, where we vary prices, ratings, and psychological nudges, all of which are factors long known to shape human choice. We find that agent decisions shift predictably and substantially in response, revealing that agents are strongly biased choosers even without being subject to the cognitive constraints that shape human biases. This susceptibility reveals both risk and opportunity: risk, because agentic consumers may inherit and amplify human biases; opportunity, because consumer choice provides a powerful testbed for a behavioral science of AI agents, just as it has for the study of human behavior. We release our framework as an open benchmark for rigorous, scalable evaluation of agent decision-making.