# MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games arXiv: https://arxiv.org/abs/2607.10645 ยท HTML: https://mlsanity.com/paper/2607.10645/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. - Builds on 38 of its 56 references that are in the map. - First posted on arXiv 12 Jul 2026. ## Placed in - LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/) ## Builds on (most cited first, 30 of 38) - [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017) - [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023) - [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022) - [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023) - [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023) - [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020) - [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020) - [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023) - [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023) - [Representation Learning for Dynamic Graphs: A Survey](https://mlsanity.com/paper/1905.11485/) (2019) - [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023) - [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023) - [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023) - [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022) - [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023) - [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023) - [Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game](https://mlsanity.com/paper/2310.18940/) (2023) - [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024) - [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023) - [Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs](https://mlsanity.com/paper/2111.13654/) (2021) - [Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4](https://mlsanity.com/paper/2309.17277/) (2023) - [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025) - [LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay](https://mlsanity.com/paper/2310.14985/) (2023) - [Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models](https://mlsanity.com/paper/2304.13835/) (2023) - [Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests](https://mlsanity.com/paper/2310.20320/) (2023) - [Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Models](https://mlsanity.com/paper/2308.01404/) (2023) - [Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction](https://mlsanity.com/paper/2407.13943/) (2024) - [Finding Friend and Foe in Multi-Agent Games](https://mlsanity.com/paper/1906.02330/) (2019) - [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023) - [Standards for Belief Representations in LLMs](https://mlsanity.com/paper/2405.21030/) (2024) ## Built on by (most cited first, 2 of 2) - [Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game](https://mlsanity.com/paper/2607.28146/) (2026) - [Replication Without Persistence in Hosted LLMs: Measurement Sensitivity in Action-Time Belief Evaluation](https://mlsanity.com/paper/2609.22478/) (2026) ## Abstract (arXiv) An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. It distinguishes whether an agent lost because it misread the game or because it failed to act on a correct assessment, a distinction that is invisible from outcomes and dialogue transcripts alone. After every public utterance, each agent privately answers structured probe questions whose responses never re-enter the game and are scored against the ground truth known to the engine. An interactive visualizer replays games from the perspective of an individual agent's beliefs, displays timeline-aligned accuracy and calibration, and supports counterfactual replay from any recorded step. In a case study across two model families comprising tens of thousands of parsed probe responses, we find that stated confidence is poorly calibrated, agents overestimate how often they are suspected by a factor of 1.5, and single-vote counterfactual replays rarely change game outcomes: outcome flips occur primarily when the agent had already formed a correct belief state, whereas decisions made under an incorrect model of the world remain largely unchanged under resampling. The engine, visualizer, recorded games, and counterfactual replay corpus are released under an open-source licence. Code: https://github.com/karpovilia/mafiascope. Live demo: https://karpovilia.github.io/mafiascope/. Screencast: https://vimeo.com/1208920221.