mlsanityData to 7 Oct 2026

Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals

arXiv 2301.01743 ↗First posted 1 Jan 2023cs.AI cs.CLcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

New chat AI applications like ChatGPT offer an advanced understanding of question context and memory across multi-step tasks, such that experiments can test its deductive reasoning. This paper proposes a multi-role and multi-step challenge, where ChatGPT plays the classic twenty-questions game but innovatively switches roles from the questioner to the answerer. The main empirical result establishes that this generation of chat applications can guess random object names in fewer than twenty questions (average, 12) and correctly guess 94% of the time across sixteen different experimental setups. The research introduces four novel cases where the chatbot fields the questions, asks the questions, both question-answer roles, and finally tries to guess appropriate contextual emotions. One task that humans typically fail but trained chat applications complete involves playing bilingual games of twenty questions (English answers to Spanish questions). Future variations address direct problem-solving using a similar inquisitive format to arrive at novel outcomes deductively, such as patentable inventions or combination thinking. Featured applications of this dialogue format include complex protein designs, neuroscience metadata, and child development educational materials.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 33 references that are in the map. First posted on arXiv 1 Jan 2023.

Builds on

6 of 33 references.

References · 6

  1. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  2. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  3. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  4. 2021Block-Skim: Efficient Question Answering for Transformercited by 16
  5. 2021More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answeringcited by 8
  6. 2022COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Taskscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024Empirical Study of Symmetrical Reasoning in Conversational Chatbots
  2. 2023InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent
  3. 2025Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course
  4. 2023ChatGPT for Robotics: Design Principles and Model Abilities
  5. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
  6. 2023Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
  7. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games
  8. 2024LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ