mlsanityData to 14 Oct 2026

MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games

arXiv 2607.10645 ↗First posted 12 Jul 2026cs.CL cs.MA cs.SIcited by 2

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. It distinguishes whether an agent lost because it misread the game or because it failed to act on a correct assessment, a distinction that is invisible from outcomes and dialogue transcripts alone. After every public utterance, each agent privately answers structured probe questions whose responses never re-enter the game and are scored against the ground truth known to the engine. An interactive visualizer replays games from the perspective of an individual agent's beliefs, displays timeline-aligned accuracy and calibration, and supports counterfactual replay from any recorded step. In a case study across two model families comprising tens of thousands of parsed probe responses, we find that stated confidence is poorly calibrated, agents overestimate how often they are suspected by a factor of 1.5, and single-vote counterfactual replays rarely change game outcomes: outcome flips occur primarily when the agent had already formed a correct belief state, whereas decisions made under an incorrect model of the world remain largely unchanged under resampling. The engine, visualizer, recorded games, and counterfactual replay corpus are released under an open-source licence. Code: https://github.com/karpovilia/mafiascope. Live demo: https://karpovilia.github.io/mafiascope/. Screencast: https://vimeo.com/1208920221.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 38 of its 56 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

38 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents26 May
  2. 2025Among Us: A Sandbox for Measuring and Detecting Agentic Deception5 Apr
  3. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay23 Oct

Other references · 35

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  6. 2020Temporal Graph Networks for Deep Learning on Dynamic Graphscited by 530
  7. 2020Inductive Representation Learning on Temporal Graphscited by 462
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  9. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  10. 2019Representation Learning for Dynamic Graphs: A Surveycited by 265
  11. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  12. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
23 more
  1. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 222
  2. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  3. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  4. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  5. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  6. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  7. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  8. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  9. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 68
  10. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  11. 2023Multi-Party Chat: Conversational Agents in Group Settings with Humans and Modelscited by 54
  12. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 48
  13. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 47
  14. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 45
  15. 2019Finding Friend and Foe in Multi-Agent Gamescited by 42
  16. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  17. 2024Standards for Belief Representations in LLMscited by 30
  18. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 26
  19. 2025ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mindcited by 21
  20. 2022Putting the Con in Context: Identifying Deceptive Actors in the Game of Mafiacited by 13
  21. 2025Bayesian Social Deduction with Graph-Informed Language Modelscited by 9
  22. 2025Deceive, Detect, and Disclose: Large Language Models Play Mini-Mafiacited by 9
  23. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Gamecited by 0
  2. 2026Replication Without Persistence in Hosted LLMs: Measurement Sensitivity in Action-Time Belief Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games
  2. 2026Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
  3. 2026Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games
  4. 2024PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Games
  5. 2026Confident at the moment of action: belief miscalibration in LLM play under hidden information
  6. 2026Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing
  7. 2026Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions
  8. 2026TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs