mlsanityData to 7 Oct 2026

Interactive Critique-Revision Training for Reliable Structured LLM Generation

arXiv 2605.08327 ↗First posted 8 May 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In structured decision-making workflows such as form filling, compliance checking, and maintenance reporting, LLM outputs must be locally correct, globally consistent, and auditable against task-specific rules. Existing refinement methods often rely on heuristic debate, self-play, or LLM-generated supervision, creating a second-order assurance problem. We propose DPA-GRPO (Dual Paired-Action Group-Relative Policy Optimization), a paired-action training method for a two-player generator--verifier game with structured verifier interventions. The generator proposes outputs and may revise them when challenged; the verifier either remains silent or raises a safety assurance case (SAC) containing a claim, argument, and evidence. These SAC/no-SAC and KEEP/REVISE decisions induce paired counterfactual action groups, which DPA-GRPO uses for role-specific KL-regularized GRPO updates. We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation. Under standard stochastic-approximation assumptions, DPA-GRPO tracks the corresponding game ODE, whose isolated asymptotically stable limit points are stationary and candidate local equilibria under role-wise local optimality. Experiments on TaxCalcBench TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines across Qwen3-4B and Qwen3-8B. Training increases correct silent acceptance, reduces missed errors, and improves calibrated revision behavior, indicating gains for both generator and verifier.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 50 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

36 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA27 Oct
  2. 2025It Takes Two: Your GRPO Is Secretly DPO1 Oct
  3. 2023Let's Verify Step by Step31 May
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2025Qwen3 Technical Reportcited by 8,195
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
18 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  4. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  5. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  6. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  7. 2024Modeling Other Players with Bayesian Beliefs for Games with Incomplete Informationcited by 16
  8. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14
  9. 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
  10. 2024Large Language Models as Agents in Two-Player Gamescited by 10
  11. 2025Optimizing Prompt Sequences using Monte Carlo Tree Search for LLM-Based Optimizationcited by 9
  12. 2025Co-Evolving Agents: Learning from Failures as Hard Negativescited by 7
  13. 2026Geometry of Drifting MDPs with Path-Integral Stability Certificatescited by 7
  14. 2026Can AI Agents Agree?cited by 7
  15. 2026Structuring Value Representations via Geometric Coherence in Markov Decision Processescited by 6
  16. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  17. 2026Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamicscited by 5
  18. 2026Game of Thought: Robust Information Seeking with Large Language Models Using Game Theorycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Auditing Emergent LLM-Agent Collaboration through Cooperation-Obligation Couplingcited by 1
  2. 2026FlowEdit: Information-Theoretic Control of LLM Reasoning Flows for Ill-posed Problems Involving Conflictscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Distill and Align Decomposition for Enhanced Claim Verification
  2. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation
  3. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
  4. 2026REVES: REvision and VErification--Augmented Training for Test-Time Scaling
  5. 2026LLM-as-a-Verifier: A General-Purpose Verification Framework
  6. 2026Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents
  7. 2026CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
  8. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning