# CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

arXiv: https://arxiv.org/abs/2604.09746 · HTML: https://mlsanity.com/paper/2604.09746/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Learning diverse attacks on large language models for robust red-teaming and safety tuning](https://mlsanity.com/paper/2405.18540/) (2024)
- [Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models](https://mlsanity.com/paper/2407.07086/) (2024)
- [Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!](https://mlsanity.com/paper/2504.09762/) (2025)
- [Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters](https://mlsanity.com/paper/2602.10604/) (2026)
- [Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents](https://mlsanity.com/paper/2305.02412/) (2023)
- [Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark](https://mlsanity.com/paper/2401.03991/) (2024)
- [Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions](https://mlsanity.com/paper/2406.19236/) (2024)
- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [Language-Driven Opinion Dynamics in Agent-Based Simulations with LLMs](https://mlsanity.com/paper/2502.19098/) (2025)
- [CoP: Agentic Red-teaming for Large Language Models using Composition of Principles](https://mlsanity.com/paper/2506.00781/) (2025)
- [When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR)](https://mlsanity.com/paper/2504.00374/) (2025)
- [Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment](https://mlsanity.com/paper/2411.11731/) (2024)
- [MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models](https://mlsanity.com/paper/2507.20395/) (2025)
- [Learning to Influence Human Behavior with Offline Reinforcement Learning](https://mlsanity.com/paper/2303.02265/) (2023)

## Built on by (most cited first, 2 of 2)

- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions](https://mlsanity.com/paper/2608.30428/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.