# Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals

arXiv: https://arxiv.org/abs/2301.01743 · HTML: https://mlsanity.com/paper/2301.01743/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 1 Jan 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 6 of 6)

- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)
- [A Survey for Efficient Open Domain Question Answering](https://mlsanity.com/paper/2211.07886/) (2022)
- [Block-Skim: Efficient Question Answering for Transformer](https://mlsanity.com/paper/2112.08560/) (2021)
- [More Than Reading Comprehension: A Survey on Datasets and Metrics of Textual Question Answering](https://mlsanity.com/paper/2109.12264/) (2021)
- [COOL, a Context Outlooker, and its Application to Question Answering and other Natural Language Processing Tasks](https://mlsanity.com/paper/2204.09593/) (2022)

## Built on by (most cited first, 3 of 3)

- [Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models](https://mlsanity.com/paper/2402.03271/) (2024)
- [Conformal Information Pursuit for Interactively Guiding Large Language Models](https://mlsanity.com/paper/2507.03279/) (2025)
- [GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models](https://mlsanity.com/paper/2509.19593/) (2025)

## Abstract (arXiv)

New chat AI applications like ChatGPT offer an advanced understanding of question context and memory across multi-step tasks, such that experiments can test its deductive reasoning. This paper proposes a multi-role and multi-step challenge, where ChatGPT plays the classic twenty-questions game but innovatively switches roles from the questioner to the answerer. The main empirical result establishes that this generation of chat applications can guess random object names in fewer than twenty questions (average, 12) and correctly guess 94% of the time across sixteen different experimental setups. The research introduces four novel cases where the chatbot fields the questions, asks the questions, both question-answer roles, and finally tries to guess appropriate contextual emotions. One task that humans typically fail but trained chat applications complete involves playing bilingual games of twenty questions (English answers to Spanish questions). Future variations address direct problem-solving using a similar inquisitive format to arrive at novel outcomes deductively, such as patentable inventions or combination thinking. Featured applications of this dialogue format include complex protein designs, neuroscience metadata, and child development educational materials.