# Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

arXiv: https://arxiv.org/abs/1705.07830 · HTML: https://mlsanity.com/paper/1705.07830/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2017.

## Built on by (most cited first, 30 of 44)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [BERTese: Learning to Speak to BERT](https://mlsanity.com/paper/2103.05327/) (2021)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning](https://mlsanity.com/paper/2112.08558/) (2021)
- [Chain-of-Retrieval Augmented Generation](https://mlsanity.com/paper/2501.14342/) (2025)
- [ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations](https://mlsanity.com/paper/2212.10409/) (2022)
- [Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering](https://mlsanity.com/paper/2106.11575/) (2021)
- [Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation](https://mlsanity.com/paper/2401.07382/) (2024)
- [Boosting Search Engines with Interactive Agents](https://mlsanity.com/paper/2109.00527/) (2021)
- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [Reinforced Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2210.15777/) (2022)
- [Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2105.04850/) (2021)
- [SCAR: Shapley Credit Assignment for More Efficient RLHF](https://mlsanity.com/paper/2505.20417/) (2025)
- [Improving Generalization in Visual Reasoning via Self-Ensemble](https://mlsanity.com/paper/2410.20883/) (2024)
- [An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation](https://mlsanity.com/paper/1908.10835/) (2019)
- [ALaRM: Align Language Models via Hierarchical Rewards Modeling](https://mlsanity.com/paper/2403.06754/) (2024)
- [Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language](https://mlsanity.com/paper/2205.06111/) (2022)
- [Can Question Rewriting Help Conversational Question Answering?](https://mlsanity.com/paper/2204.06239/) (2022)
- [Robust Training for Conversational Question Answering Models with Reinforced Reformulation Generation](https://mlsanity.com/paper/2310.13505/) (2023)
- [Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancy](https://mlsanity.com/paper/2511.12920/) (2025)
- [Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization](https://mlsanity.com/paper/2110.04741/) (2021)
- [Zero-Shot Retrieval with Search Agents and Hybrid Environments](https://mlsanity.com/paper/2209.15469/) (2022)
- [Decoding a Neural Retriever's Latent Space for Query Suggestion](https://mlsanity.com/paper/2210.12084/) (2022)
- [The Curious Language Model: Strategic Test-Time Information Acquisition](https://mlsanity.com/paper/2506.09173/) (2025)
- [Query Decomposition for RAG: Balancing Exploration-Exploitation](https://mlsanity.com/paper/2510.18633/) (2025)
- [Conversational Question Answering with Reformulations over Knowledge Graph](https://mlsanity.com/paper/2312.17269/) (2023)
- [Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs](https://mlsanity.com/paper/2605.07153/) (2026)
- [High-Quality Diversification for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2106.00891/) (2021)

## Abstract (arXiv)

We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.