mlsanityData to 14 Oct 2026

Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

arXiv 2608.20362 ↗First posted 17 Jun 2026cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this assumption fails in multilingual settings: an exact-match verifier turns format and script variation into language-dependent false-negative reward noise. We introduce a reusable protocol for auditing multilingual RLVR rewards: a verifier-robustness suite, a rollout-diagnosis procedure, and language-conditioned reward-error metrics for Japanese, English, and Chinese answers. On MGSM rollouts with k=8, the exact-match proxy rejects trusted-correct answers at sharply different rates by language across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct; for Qwen3-8B, the false-negative rate reaches 0.642 on JP against 0.122 on EN and 0.073 on CN. A plain-numeric probe localizes the mechanism to the final-answer interface: an interface model drives reward-error VLB to zero while the residual accuracy gap is unchanged. We then expose a cross-lingual selection bottleneck: on MGSM250 rollouts, a target-local aggregation rule using no trusted labels closes 55-78% of the average selection gap, and over 95% of repairs require genuine cross-lingual support. The bottleneck replicates on a 483-problem MATH-500 set. A controlled training audit shows that rule-GRPO raises trusted accuracy while the reward-error VLB stays high. The unifying message is operational: multilingual RLVR rewards should be audited by language and by answer interface before they are optimized.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 17 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

15 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking16 Apr
  2. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning21 May
  3. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning20 May

Other references · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  5. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 45
  6. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  7. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 5
  8. 2026Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Rewardcited by 3
  9. 2026Cross-lingual Self-Consistency for Multilingual Reasoning with Language Modelscited by 3
  10. 2025Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Modelscited by 2
  11. 2026MATH-PT: A Math Reasoning Benchmark for European and Brazilian Portuguesecited by 2
  12. 2026Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewardscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR
  2. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
  3. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  4. 2026Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR
  5. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  6. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  8. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR