mlsanityData to 7 Oct 2026

Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation

arXiv 2310.01320 ↗First posted 2 Oct 2023cs.AI cs.CL cs.CYcited by 102

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent breakthroughs in large language models (LLMs) have brought remarkable success in the field of LLM-as-Agent. Nevertheless, a prevalent assumption is that the information processed by LLMs is consistently honest, neglecting the pervasive deceptive or misleading information in human society and AI-generated content. This oversight makes LLMs susceptible to malicious manipulations, potentially resulting in detrimental outcomes. This study utilizes the intricate Avalon game as a testbed to explore LLMs' potential in deceptive environments. Avalon, full of misinformation and requiring sophisticated logic, manifests as a "Game-of-Thoughts". Inspired by the efficacy of humans' recursive thinking and perspective-taking in the Avalon game, we introduce a novel framework, Recursive Contemplation (ReCon), to enhance LLMs' ability to identify and counteract deceptive information. ReCon combines formulation and refinement contemplation processes; formulation contemplation produces initial thoughts and speech, while refinement contemplation further polishes them. Additionally, we incorporate first-order and second-order perspective transitions into these processes respectively. Specifically, the first-order allows an LLM agent to infer others' mental states, and the second-order involves understanding how others perceive the agent's mental state. After integrating ReCon with different LLMs, extensive experiment results from the Avalon game indicate its efficacy in aiding LLMs to discern and maneuver around deceptive information without extra fine-tuning and data. Finally, we offer a possible explanation for the efficacy of ReCon and explore the current limitations of LLMs in terms of safety, reasoning, speaking style, and format, potentially furnishing insights for subsequent research.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 48 of its 62 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

48 of 62 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2020Learning to summarize from human feedbackcited by 2,562
36 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  8. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  9. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  12. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  13. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  14. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  15. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  16. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  17. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  18. 2023Playing repeated games with Large Language Modelscited by 213
  19. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  20. 2023Large Language Models as General Pattern Machinescited by 196
  21. 2023Augmenting Language Models with Long-Term Memorycited by 186
  22. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  23. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  24. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  25. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  26. 2023Model evaluation for extreme riskscited by 167
  27. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  28. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  29. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  30. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  31. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  32. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  33. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  34. 2019Finding Friend and Foe in Multi-Agent Gamescited by 42
  35. 2023Let's Do a Thought Experiment: Using Counterfactuals to Improve Moral Reasoningcited by 20
  36. 2023Playing the Werewolf game with artificial intelligence for language understandingcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 39; 2025: 37; 2026: 21.

102 papers.

Sort:

By area:

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  3. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  4. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  6. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 118
  9. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  10. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  11. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  12. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
38 more
  1. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  2. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  3. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  4. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  5. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 56
  6. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  7. 2025Do as We Do, Not as You Think: the Conformity of Large Language Modelscited by 48
  8. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  9. 2023Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approachcited by 38
  10. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 30
  11. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  12. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  13. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  14. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  15. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systemscited by 26
  16. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 25
  17. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  18. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  19. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 22
  20. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  21. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  22. 2024Iterative Experience Refinement of Software-Developing Agentscited by 16
  23. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 16
  24. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigationcited by 15
  25. 2024Large Language Models Fall Short: Understanding Complex Relationships in Detective Narrativescited by 14
  26. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 14
  27. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  28. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  29. 2025Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimizationcited by 13
  30. 2025Accelerating Earth Science Discovery via Multi-Agent LLM Systemscited by 13
  31. 2024Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Gamecited by 12
  32. 2024A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Modelscited by 12
  33. 2025BookWorld: From Novels to Interactive Agent Societies for Creative Story Generationcited by 12
  34. 2025MultiMind: Enhancing Werewolf Agents with Multimodal Reasoning and Theory of Mindcited by 11
  35. 2024PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Gamescited by 10
  36. 2024Large Language Model Sentinel: LLM Agent for Adversarial Purificationcited by 10
  37. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  38. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing
  2. 2026LieCraft: A Multi-Agent Framework for Evaluating Deceptive Capabilities in Language Models
  3. 2025When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Models
  4. 2025LLMs as Deceptive Agents: How Role-Based Prompting Induces Semantic Ambiguity in Puzzle Tasks
  5. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deception
  6. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts
  7. 2026Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models
  8. 2026Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games