Assessing Rule Adherence of LLM Adjudicators in Call of Cthulhu TRPG
Not on the map. Why?
Abstract (arXiv)
As LLMs are increasingly deployed as autonomous adjudicators in games such as Call of Cthulhu (CoC), robust rule adherence becomes critical when user intent conflicts with system rules. However, as these models are trained to be helpful and compliant, they may be vulnerable to a class of manipulations we term Rhetorical Injection, where adversarial users exploit narrative framing techniques such as pseudo-logical reasoning and authoritative coercion to bypass adjudication logic. We present CoC-Seduce, a multi-agent adversarial benchmark built on CoC, a Tabletop Role-Playing Game (TRPG) in which rules are explicit about which risky actions require adjudication, yet interaction remains entirely in natural language. Three LLMs, i.e., GPT-5.4, Claude Sonnet 4.6, Gemini 3.5 Flash, serve as adversarial generators producing 5,376 samples across 4 world settings and 16 skill categories. We then benchmark 22 target adjudicators against this corpus. Evaluation across 22 models reveals that neither newer releases nor explicit reasoning reliably confer adjudication robustness, that Pseudo-Logic framing is the most effective rhetorical style, and that the world setting, including culturally distant ones, has only a modest effect. Project page: https://github.com/answerrtx/CoC-Seduce.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 43 references that are in the map. First posted on arXiv 2 Jul 2026.
Builds on
15 of 43 references.
References · 15
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
- 2023FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Informationcited by 18
- 2024RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Wordscited by 4
- 2024You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Callingcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
- 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
- 2024Self-playing Adversarial Language Game Enhances LLM Reasoning
- 2026Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agents
- 2025LLMs as Deceptive Agents: How Role-Based Prompting Induces Semantic Ambiguity in Puzzle Tasks
- 2026High-quality generation of dynamic game content via small language models: A proof of concept
- 2026The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?
- 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games