# When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

arXiv: https://arxiv.org/abs/2610.00202 · HTML: https://mlsanity.com/paper/2610.00202/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers](https://mlsanity.com/paper/2510.00915/) (2025)
- [LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking](https://mlsanity.com/paper/2604.15149/) (2026)
- [Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias](https://mlsanity.com/paper/2606.19544/) (2026)
- [From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning](https://mlsanity.com/paper/2505.22203/) (2025)
- [Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text](https://mlsanity.com/paper/2601.22975/) (2026)
- [Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR](https://mlsanity.com/paper/2605.02909/) (2026)
- [Before the Model Learns the Bug:Fuzzing RLVR Verifiers](https://mlsanity.com/paper/2606.01066/) (2026)
- [Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applications](https://mlsanity.com/paper/2502.00808/) (2025)
- [Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design](https://mlsanity.com/paper/2603.12826/) (2026)
- [Do Large Language Models Plan Answer Positions? Position Bias in Multiple-Choice Question Generation](https://mlsanity.com/paper/2605.01846/) (2026)

## Abstract (arXiv)

Several recent pipelines build RLVR training data by masking a span of real corpus text and asking a language model to invent plausible wrong answers around it. The correct option is therefore genuine human prose; every distractor is synthetic. Correctness and provenance become entangled, and a policy could in principle learn the second instead of the first. We audit that possibility in GooseReason-0.7M. First we ask whether the asymmetry is visible at all: a classifier reading only five surface statistics (never the meaning) reaches AUROC 0.562 over 315,499 options, barely above chance. The aggregate hides something, though. Code sits at 0.416, below chance, and manual inspection explains why: code distractors turn out to be single-operator mutations of the gold answer rather than freely written alternatives, so the two classes are nearly identical by construction. Detecting a signal is not the same as showing a model uses it, so we then run an intervention. We build a paraphrase-matched control corpus, hold training-set size identical across arms, and train two policies under one fixed budget. The exploitation gap does not favour the unmodified-data arm: 0.021 against 0.027 for the control. Under our budget, in other words, a detectable artifact went unexploited. We think that dissociation, along with the domain-specific construction finding, is worth knowing for anyone curating corpora of this kind, and we release the audit as a mostly CPU-only protocol.