# GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning

arXiv: https://arxiv.org/abs/2505.22661 · HTML: https://mlsanity.com/paper/2505.22661/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 26 of 26)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Medical Hallucinations in Foundation Models and Their Impact on Healthcare](https://mlsanity.com/paper/2503.05777/) (2025)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [AutoEval Done Right: Using Synthetic Data for Model Evaluation](https://mlsanity.com/paper/2403.07008/) (2024)
- [GameArena: Evaluating LLM Reasoning through Live Computer Games](https://mlsanity.com/paper/2412.06394/) (2024)
- [CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery](https://mlsanity.com/paper/2406.08587/) (2024)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games](https://mlsanity.com/paper/2310.01468/) (2023)
- [Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models](https://mlsanity.com/paper/2410.20745/) (2024)
- [xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation](https://mlsanity.com/paper/2405.11874/) (2024)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [Automatic Dataset Construction (ADC): Sample Collection, Data Curation, and Beyond](https://mlsanity.com/paper/2408.11338/) (2024)
- [TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles](https://mlsanity.com/paper/2410.05262/) (2024)
- [Financial Knowledge Large Language Model](https://mlsanity.com/paper/2407.00365/) (2024)
- [A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Models](https://mlsanity.com/paper/2504.02793/) (2025)

## Built on by (most cited first, 3 of 3)

- [A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench](https://mlsanity.com/paper/2507.08491/) (2025)
- [When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMs](https://mlsanity.com/paper/2510.15421/) (2025)
- [WordPolo: Evaluating Language Models Through Iterative Semantic Feedback](https://mlsanity.com/paper/2609.19006/) (2026)

## Abstract (arXiv)

The evaluation of large language models (LLMs) has traditionally relied on static benchmarks, a paradigm that poses two major limitations: (1) predefined test sets lack adaptability to diverse application domains, and (2) standardized evaluation protocols often fail to capture fine-grained assessments of domain-specific knowledge and contextual reasoning abilities. To overcome these challenges, we propose GuessArena, an adaptive evaluation framework grounded in adversarial game-based interactions. Inspired by the interactive structure of the Guess Who I Am? game, our framework seamlessly integrates dynamic domain knowledge modeling with progressive reasoning assessment to improve evaluation fidelity. Empirical studies across five vertical domains-finance, healthcare, manufacturing, information technology, and education-demonstrate that GuessArena effectively distinguishes LLMs in terms of domain knowledge coverage and reasoning chain completeness. Compared to conventional benchmarks, our method provides substantial advantages in interpretability, scalability, and scenario adaptability.