mlsanityData to 13 Oct 2026

Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

arXiv 1705.07830 ↗First posted 22 May 2017cs.CL cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

Key facts. Cited by 44 papers in the map (data through 13 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 9; 2022: 10; 2023: 7; 2024: 7; 2025: 8; 2026: 2.

44 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,229
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,063
  3. 2023Augmented Language Models: a Surveycited by 403
  4. 2021BERTese: Learning to Speak to BERTcited by 82
  5. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  6. 2021CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learningcited by 68
  7. 2025Chain-of-Retrieval Augmented Generationcited by 49
  8. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  9. 2021Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answeringcited by 27
  10. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  11. 2021Boosting Search Engines with Interactive Agentscited by 19
  12. 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
32 more
  1. 2022Reinforced Question Rewriting for Conversational Question Answeringcited by 19
  2. 2021Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphscited by 18
  3. 2024Improving Generalization in Visual Reasoning via Self-Ensemblecited by 15
  4. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
  5. 2019An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generationcited by 11
  6. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
  7. 2022Asking for Knowledge: Training RL Agents to Query External Knowledge Using Languagecited by 10
  8. 2022Can Question Rewriting Help Conversational Question Answering?cited by 8
  9. 2023Robust Training for Conversational Question Answering Models with Reinforced Reformulation Generationcited by 7
  10. 2025Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancycited by 7
  11. 2021Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarizationcited by 6
  12. 2022Zero-Shot Retrieval with Search Agents and Hybrid Environmentscited by 6
  13. 2022Decoding a Neural Retriever's Latent Space for Query Suggestioncited by 5
  14. 2025The Curious Language Model: Strategic Test-Time Information Acquisitioncited by 5
  15. 2025Query Decomposition for RAG: Balancing Exploration-Exploitationcited by 5
  16. 2023Conversational Question Answering with Reformulations over Knowledge Graphcited by 4
  17. 2021High-Quality Diversification for Task-Oriented Dialogue Systemscited by 3
  18. 2023Answering Unanswered Questions through Semantic Reformulations in Spoken QAcited by 3
  19. 2024Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncologycited by 3
  20. 2024Improving Long Text Understanding with Knowledge Distilled from Summarization Modelcited by 3
  21. 2026Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMscited by 3
  22. 2022Can Open Domain Question Answering Systems Answer Visual Knowledge Questions?cited by 2
  23. 2022A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOScited by 2
  24. 2023Successor Features for Efficient Multisubject Controlled Text Generationcited by 2
  25. 2022Query Expansion and Entity Weighting for Query Reformulation Retrieval in Voice Assistant Systemscited by 1
  26. 2023Coherence and Diversity through Noise: Self-Supervised Paraphrase Generation via Structure-Aware Denoisingcited by 1
  27. 2024Reinforcement Learning for Conversational Question Answering over Knowledge Graphcited by 1
  28. 2024PL-FSCIL: Harnessing the Power of Prompts for Few-Shot Class-Incremental Learningcited by 1
  29. 2025Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Predictioncited by 0
  30. 2025EAPO: Enhancing Policy Optimization with On-Demand Expert Assistancecited by 0
  31. 2025Towards better dense rewards in Reinforcement Learning Applicationscited by 0
  32. 2026Harness-G: A Graph-Structured Harness for Search Agentscited by 0