mlsanityData to 7 Oct 2026

HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards

arXiv 2608.06012 ↗First posted 6 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Search-agent rewards mix answer quality, citation grounding, tool cost, and anti-hacking terms; a high score therefore need not imply that cited evidence was retrieved, and added penalties can cancel. We introduce HERALD, an offline audit that applies exact same-question interventions, separates candidate-visible from oracle information, and enumerates detector contracts before policy optimization. On four Qwen3-8B pools from HotpotQA, 2WikiMultiHopQA, and MuSiQue, R0 rejects search deletion and fake IDs, but a label-free citation-laundering attack succeeds. A complete 23 ablation identifies targeted strengthening of L---citing a corpus passage absent from the retrieved evidence---as the observed inclusion-minimal repair: R[L] has zero empirical ASR with a 0.50% one-sided cluster upper bound. The gap persists across pool rules, a visible BM25 attacker, and four models; broader hardening remains vulnerable when the attack removes an oracle support-ID penalty. Under strict 5M-token matched training evaluated on 256 paired questions per benchmark, R[L] meets the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue. Equal-suite citation precision and support recall improve by 2.02 and 1.46 points, unsupported citations fall by 1.69, and laundering attackability falls on 2Wiki and MuSiQue. Natural L is not reduced, and the detector appears in only 18 of 58,368 training trajectories. HERALD thus separates robust scoring, sparse learning signal, and policy transfer.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

26 of 32 references.

References · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  10. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  11. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  12. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
14 more
  1. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  2. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  4. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  5. 2025One Token to Fool LLM-as-a-Judgecited by 56
  6. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
  7. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 25
  8. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 24
  9. 2025DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learningcited by 14
  10. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  11. 2026Counterfactual Trace Auditing of LLM Agent Skillscited by 8
  12. 2024RATE: Causal Explainability of Reward Models with Imperfect Counterfactualscited by 6
  13. 2025Proof-of-Use: Mitigating Tool-Call Hacking in Deep Research Agentscited by 3
  14. 2026STAMP: Provenance-Guided Credit Assignment for Deep Search Agentscited by 3

Built on by

Not cited yet.