Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
New chat AI applications like ChatGPT offer an advanced understanding of question context and memory across multi-step tasks, such that experiments can test its deductive reasoning. This paper proposes a multi-role and multi-step challenge, where ChatGPT plays the classic twenty-questions game but innovatively switches roles from the questioner to the answerer. The main empirical result establishes that this generation of chat applications can guess random object names in fewer than twenty questions (average, 12) and correctly guess 94% of the time across sixteen different experimental setups. The research introduces four novel cases where the chatbot fields the questions, asks the questions, both question-answer roles, and finally tries to guess appropriate contextual emotions. One task that humans typically fail but trained chat applications complete involves playing bilingual games of twenty questions (English answers to Spanish questions). Future variations address direct problem-solving using a similar inquisitive format to arrive at novel outcomes deductively, such as patentable inventions or combination thinking. Featured applications of this dialogue format include complex protein designs, neuroscience metadata, and child development educational materials.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 33 references that are in the map. First posted on arXiv 1 Jan 2023.
Builds on
6 of 33 references.
References · 6
- 2022Reasoning with Language Model Prompting: A Surveycited by 251
- 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
- 2022A Survey for Efficient Open Domain Question Answeringcited by 20
- 2021Block-Skim: Efficient Question Answering for Transformercited by 16
- 2021More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answeringcited by 8
- 2022COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Taskscited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.
- 2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Modelscited by 37
- 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
- 2025GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Modelscited by 1
- 2025GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models23 Sep 2025
- 2025Conformal Information Pursuit for Interactively Guiding Large Language Models4 Jul 2025
- 2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models5 Feb 2024
Similar, not cited
Close in wording, no citation either way
- 2024Empirical Study of Symmetrical Reasoning in Conversational Chatbots
- 2023InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent
- 2025Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course
- 2023ChatGPT for Robotics: Design Principles and Model Abilities
- 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
- 2023Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
- 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
- 2024LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ