# What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA

arXiv: https://arxiv.org/abs/2605.25988 · HTML: https://mlsanity.com/paper/2605.25988/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 23 of 23)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Lessons from Natural Language Inference in the Clinical Domain](https://mlsanity.com/paper/1808.06752/) (2018)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [SciFive: a text-to-text transformer model for biomedical literature](https://mlsanity.com/paper/2106.03598/) (2021)
- [MedBrowseComp: Benchmarking Medical Deep Research and Computer Use](https://mlsanity.com/paper/2505.14963/) (2025)
- [MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Framework](https://mlsanity.com/paper/2508.14880/) (2025)
- [Step-by-Step Fact Verification System for Medical Claims with Explainable Reasoning](https://mlsanity.com/paper/2502.14765/) (2025)
- [LiveQA: A Question Answering Dataset over Sports Live](https://mlsanity.com/paper/2010.00526/) (2020)
- [Examining and Adapting Time for Multilingual Classification via Mixture of Temporal Experts](https://mlsanity.com/paper/2502.08825/) (2025)
- [Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memory](https://mlsanity.com/paper/2605.03228/) (2026)
- [EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models](https://mlsanity.com/paper/2603.19532/) (2026)
- [Improving Reliability and Explainability of Medical Question Answering through Atomic Fact Checking in Retrieval-Augmented LLMs](https://mlsanity.com/paper/2505.24830/) (2025)
- [Deep Neural Network Architectures for Electrocardiogram Classification: A Comprehensive Evaluation](https://mlsanity.com/paper/2602.17701/) (2026)
- [Knowledge-driven Augmentation and Retrieval for Integrative Temporal Adaptation](https://mlsanity.com/paper/2604.22098/) (2026)

## Abstract (arXiv)

Medical RAG needs evidence-grounded claims, so plugging a claim-level NLI checker into retrieval-augmented RL is intuitive. textbfWe find that the checker's output distribution during training, not its held-out accuracy, decides whether it provides trainable gradient. We compare four NLI checker back-ends as process rewards inside a GRPO-trained medical RAG agent (Qwen2.5-7B, replicated on Qwen3-4B and Llama-3.1-8B) across four held-out medical QA benchmarks. Three diagnostic findings emerge. (i) Signal collapse is log-prob-specific: LLM log-probability scoring labels over 97% of claims neutral -- collapsing the RL gradient to zero -- while a calibrated MedNLI classifier scores the same pairs non-degenerately. (ii) Moderate signal beats strong signal on answer quality: a strong proprietary checker triggers a three-step reward-hacking cascade -- ultra-short answers, search avoidance, language collapse -- so a moderate-signal local classifier trains a higher-quality model (+12% BERTScore over zero-shot, no GPT dependency). (iii) Signal strength is policy-dependent: the same checker registers as moderate on one policy but strong on another without triggering the cascade end-state. We frame these as boundary conditions for verifier-as-reward systems.