# Do LLMs Strategically Reveal, Conceal, and Infer Information? A Theoretical and Empirical Analysis in The Chameleon Game arXiv: https://arxiv.org/abs/2501.19398 ยท HTML: https://mlsanity.com/paper/2501.19398/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 21 of its 58 references that are in the map. - First posted on arXiv 31 Jan 2025. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 21 of 21) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022) - [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023) - [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023) - [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023) - [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023) - [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023) - [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023) - [Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation](https://mlsanity.com/paper/2309.17234/) (2023) - [Truth is Universal: Robust Detection of Lies in LLMs](https://mlsanity.com/paper/2407.12831/) (2024) - [Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories](https://mlsanity.com/paper/2406.00034/) (2024) - [Can large language models explore in-context?](https://mlsanity.com/paper/2403.15371/) (2024) - [Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4](https://mlsanity.com/paper/2309.17277/) (2023) - [Game-theoretic LLM: Agent Workflow for Negotiation Games](https://mlsanity.com/paper/2411.05990/) (2024) - [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023) - [Welfare Diplomacy: Benchmarking Language Model Cooperation](https://mlsanity.com/paper/2310.08901/) (2023) - [Steering Language Models with Game-Theoretic Solvers](https://mlsanity.com/paper/2402.01704/) (2024) - [Controlling Large Language Models Through Concept Activation Vectors](https://mlsanity.com/paper/2501.05764/) (2025) - [Generative AI in Mafia-like Game Simulation](https://mlsanity.com/paper/2309.11672/) (2023) ## Built on by (most cited first, 5 of 5) - [LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents](https://mlsanity.com/paper/2601.06973/) (2026) - [Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions](https://mlsanity.com/paper/2605.00226/) (2026) - [Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing](https://mlsanity.com/paper/2605.10794/) (2026) - [Agentmandering: A Game-Theoretic Framework for Fair Redistricting via Large Language Model Agents](https://mlsanity.com/paper/2511.04076/) (2025) - [How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling](https://mlsanity.com/paper/2610.02542/) (2026) ## Abstract (arXiv) Large language model-based (LLM-based) agents have become common in settings that include non-cooperative parties. In such settings, agents' decision-making needs to conceal information from their adversaries, reveal information to their cooperators, and infer information to identify the other agents' characteristics. To investigate whether LLMs have these information control and decision-making capabilities, we make LLM agents play the language-based hidden-identity game, The Chameleon. In this game, a group of non-chameleon agents who do not know each other aim to identify the chameleon agent without revealing a secret. The game requires the aforementioned information control capabilities both as a chameleon and a non-chameleon. We begin with a theoretical analysis for a spectrum of strategies, from concealing to revealing, and provide bounds on the non-chameleons' winning probability. The empirical results with GPT, Gemini 2.5 Pro, Llama 3.1, and Qwen3 models show that while non-chameleon LLM agents identify the chameleon, they fail to conceal the secret from the chameleon, and their winning probability is far from the levels of even trivial strategies. Based on these empirical results and our theoretical analysis, we deduce that LLM-based agents may reveal excessive information to agents of unknown identities. Interestingly, we find that, when instructed to adopt an information-revealing level, this level is linearly encoded in the LLM's internal representations. While the instructions alone are often ineffective at making non-chameleon LLMs conceal, we show that steering the internal representations in this linear direction directly can reliably induce concealing behavior.