Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fixed library of prompt attacks to adaptive strategies that steer agents toward unfavorable outcomes. We propose profit-driven red teaming, a stress-testing protocol that replaces handcrafted attacks with a learned opponent trained to maximize its profit using only scalar outcome feedback. The protocol requires no LLM-as-judge scoring, attack labels, or attack taxonomy, and is designed for structured settings with auditable outcomes. We instantiate it in a lean arena of four canonical economic interactions, which provide a controlled testbed for adaptive exploitability. In controlled experiments, agents that appear strong against static baselines become consistently exploitable under profit-optimized pressure, and the learned opponent discovers probing, anchoring, and deceptive commitments without explicit instruction. We then distill exploit episodes into concise prompt rules for the agent, which make most previously observed failures ineffective and substantially improve target performance. These results suggest that profit-driven red-team data can provide a practical route to improving robustness in structured agent settings with auditable outcomes.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 24 references that are in the map. First posted on arXiv 21 Mar 2026.
Builds on
22 of 24 references.
Closest ancestors · 1
Same direction, newest first
Other references · 21
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
- 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
- 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
- 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
- 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
9 more
- 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 100
- 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 96
- 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
- 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 67
- 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Modelscited by 59
- 2024Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Methodcited by 48
- 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
- 2024Defending Jailbreak Prompts via In-Context Adversarial Gamecited by 29
- 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
- 2026AgenticRed: Evolving Agentic Systems for Red-Teaming
- 2026OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution
- 2026CART: Closed-Loop Adaptive Red Teaming for Large Language Models
- 2026MAStrike: Shapley-Guided Collusive Red-Teaming on Multi-Agent Systems
- 2026Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming
- 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
- 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition