# PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

arXiv: https://arxiv.org/abs/2402.12326 · HTML: https://mlsanity.com/paper/2402.12326/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 80 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Wars](https://mlsanity.com/paper/2311.17227/) (2023)
- [Agents: An Open-source Framework for Autonomous Language Agents](https://mlsanity.com/paper/2309.07870/) (2023)
- [PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations](https://mlsanity.com/paper/2307.02762/) (2023)
- [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023)
- [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021)
- [RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text](https://mlsanity.com/paper/2305.13304/) (2023)
- [Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction](https://mlsanity.com/paper/2305.02466/) (2023)
- [Turning large language models into cognitive models](https://mlsanity.com/paper/2306.03917/) (2023)
- [Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Prompting](https://mlsanity.com/paper/2310.07146/) (2023)
- [Weaver: Foundation Models for Creative Writing](https://mlsanity.com/paper/2401.17268/) (2024)
- [Knowledge-enhanced Agents for Interactive Text Games](https://mlsanity.com/paper/2305.05091/) (2023)
- [Ambient Adventures: Teaching ChatGPT on Developing Complex Stories](https://mlsanity.com/paper/2308.01734/) (2023)

## Built on by (most cited first, 28 of 28)

- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection](https://mlsanity.com/paper/2502.11448/) (2025)
- [MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series](https://mlsanity.com/paper/2405.19327/) (2024)
- [LLM-based Human Simulations Have Not Yet Been Reliable](https://mlsanity.com/paper/2501.08579/) (2025)
- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models](https://mlsanity.com/paper/2409.12106/) (2024)
- [Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation](https://mlsanity.com/paper/2502.17899/) (2025)
- [A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions](https://mlsanity.com/paper/2502.11095/) (2025)
- [PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agents](https://mlsanity.com/paper/2509.23614/) (2025)
- [Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Research](https://mlsanity.com/paper/2506.01839/) (2025)
- [Can Agents Fix Agent Issues?](https://mlsanity.com/paper/2505.20749/) (2025)
- [AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior](https://mlsanity.com/paper/2403.13433/) (2024)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [PsyDI: Towards a Personalized and Progressively In-depth Chatbot for Psychological Measurements](https://mlsanity.com/paper/2408.03337/) (2024)
- [gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling](https://mlsanity.com/paper/2504.14775/) (2025)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials](https://mlsanity.com/paper/2511.00833/) (2025)
- [Inertia in Moral and Value Judgments of Large Language Models](https://mlsanity.com/paper/2408.09049/) (2024)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulation](https://mlsanity.com/paper/2512.18244/) (2025)
- [Responsible Evaluation of AI for Mental Health](https://mlsanity.com/paper/2602.00065/) (2026)
- [Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization](https://mlsanity.com/paper/2501.16106/) (2025)
- [The Odyssey of the Fittest: Can Agents Survive and Still Be Good?](https://mlsanity.com/paper/2502.05442/) (2025)
- [Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems](https://mlsanity.com/paper/2606.07805/) (2026)
- [FurniMAS: Language-Guided Furniture Decoration using Multi-Agent System](https://mlsanity.com/paper/2507.04770/) (2025)
- [Cognitive Policy-Driven LLM for Diagnosis and Intervention of Cognitive Distortions in Emotional Support Conversation](https://mlsanity.com/paper/2604.17178/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

Psychological measurement is essential for mental health, self-understanding, and personal development. Traditional methods, such as self-report scales and psychologist interviews, often face challenges with engagement and accessibility. While game-based and LLM-based tools have been explored to improve user interest and automate assessment, they struggle to balance engagement with generalizability. In this work, we propose PsychoGAT (Psychological Game AgenTs) to achieve a generic gamification of psychological assessment. The main insight is that powerful LLMs can function both as adept psychologists and innovative game designers. By incorporating LLM agents into designated roles and carefully managing their interactions, PsychoGAT can transform any standardized scales into personalized and engaging interactive fiction games. To validate the proposed method, we conduct psychometric evaluations to assess its effectiveness and employ human evaluators to examine the generated content across various psychological constructs, including depression, cognitive distortions, and personality traits. Results demonstrate that PsychoGAT serves as an effective assessment tool, achieving statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity. Moreover, human evaluations confirm PsychoGAT's enhancements in content coherence, interactivity, interest, immersion, and satisfaction.