# When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification

arXiv: https://arxiv.org/abs/2602.11199 · HTML: https://mlsanity.com/paper/2602.11199/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Interpretation of Natural Language Rules in Conversational Machine Reading](https://mlsanity.com/paper/1809.01494/) (2018)
- [Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents](https://mlsanity.com/paper/2402.09205/) (2024)
- [API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access](https://mlsanity.com/paper/2403.01216/) (2024)
- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse-to-Fine Generation](https://mlsanity.com/paper/2601.08430/) (2026)
- [Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval](https://mlsanity.com/paper/2410.04585/) (2024)
- [Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education](https://mlsanity.com/paper/2309.14534/) (2023)
- [ClarQ: A large-scale and diverse dataset for Clarification Question Generation](https://mlsanity.com/paper/2006.05986/) (2020)
- [The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers](https://mlsanity.com/paper/2404.02806/) (2024)
- [How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation](https://mlsanity.com/paper/2503.09598/) (2025)
- [AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification](https://mlsanity.com/paper/2503.01940/) (2025)
- [Open-Retrieval Conversational Machine Reading](https://mlsanity.com/paper/2102.08633/) (2021)
- [First Ask Then Answer: A Framework Design for AI Dialogue Based on Supplementary Questioning with Large Language Models](https://mlsanity.com/paper/2508.08308/) (2025)
- [Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining](https://mlsanity.com/paper/2510.22931/) (2025)

## Built on by (most cited first, 16 of 16)

- [AgentAbstain: Do LLM Agents Know When Not to Act?](https://mlsanity.com/paper/2607.10059/) (2026)
- [Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents](https://mlsanity.com/paper/2605.09698/) (2026)
- [Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction](https://mlsanity.com/paper/2604.04325/) (2026)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents](https://mlsanity.com/paper/2604.16752/) (2026)
- [MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning](https://mlsanity.com/paper/2607.07391/) (2026)
- [FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review](https://mlsanity.com/paper/2608.25325/) (2026)
- [Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization](https://mlsanity.com/paper/2609.05258/) (2026)
- [When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain](https://mlsanity.com/paper/2609.24290/) (2026)
- [Clarification Is Not Correction: LLMs Fail to Let Go](https://mlsanity.com/paper/2609.25337/) (2026)
- [Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents](https://mlsanity.com/paper/2609.37236/) (2026)
- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)
- [DelegationBench: Measuring When AI Agents Should Ask Before Acting](https://mlsanity.com/paper/2610.05532/) (2026)
- [CLEAT: Learning Value-Guided Clarification for Interactive Agents](https://mlsanity.com/paper/2610.12655/) (2026)
- [Beyond Perfect Teammates: Multi-Agent Collaboration with Unreliable Colleagues](https://mlsanity.com/paper/2610.16117/) (2026)
- [What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds](https://mlsanity.com/paper/2610.20195/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often respond even when prompts omit critical details or include misleading information, leading to hallucinations or reinforced misconceptions. We study how to evaluate and improve LLMs' ability to decide when and what to ask for clarification without sacrificing task performance. We introduce AskBench, an interactive benchmark that converts standard QA pairs into multi-turn interactions with explicit checkpoints. A unified judge loop evaluates final answers and simulates user responses as needed. AskBench covers two settings: AskMind, with intent-deficient queries requiring clarification, and AskOverconfidence, with queries containing false premises that must be identified and corrected. We further propose rubric-guided reinforcement learning with verifier-based rewards (RLVR), which uses structured rubrics to encourage targeted clarification. Experiments show consistent improvements in accuracy, rubric adherence, and interaction efficiency, with strong generalization to unseen domains.