# Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model

arXiv: https://arxiv.org/abs/2509.25543 · HTML: https://mlsanity.com/paper/2509.25543/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains](https://mlsanity.com/paper/2503.23829/) (2025)
- [Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization](https://mlsanity.com/paper/2504.05812/) (2025)
- [INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://mlsanity.com/paper/2411.19799/) (2024)
- [RLPR: Extrapolating RLVR to General Domains without Verifiers](https://mlsanity.com/paper/2506.18254/) (2025)
- [NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions](https://mlsanity.com/paper/2502.13124/) (2025)
- [MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation](https://mlsanity.com/paper/2503.10497/) (2025)
- [HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages](https://mlsanity.com/paper/2505.11475/) (2025)
- [OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework](https://mlsanity.com/paper/2405.11143/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Tower+: Bridging Generality and Translation Specialization in Multilingual LLMs](https://mlsanity.com/paper/2506.17080/) (2025)
- [P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs](https://mlsanity.com/paper/2411.09116/) (2024)
- [From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning](https://mlsanity.com/paper/2505.22203/) (2025)

## Built on by (most cited first, 4 of 4)

- [Crosslingual On-Policy Self-Distillation for Multilingual Reasoning](https://mlsanity.com/paper/2605.09548/) (2026)
- [What Makes Good Multilingual Reasoning? Disentangling Traces with Measurable Features](https://mlsanity.com/paper/2604.04720/) (2026)
- [Soft Token Alignment for Cross-Lingual Reasoning](https://mlsanity.com/paper/2606.26466/) (2026)
- [GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings](https://mlsanity.com/paper/2608.13698/) (2026)

## Abstract (arXiv)

While reinforcement learning has advanced the reasoning abilities of Large Language Models (LLMs), these gains are largely confined to English, creating a significant performance disparity across languages. To address this, we introduce Pivot-Based Reinforcement Learning with Semantically Verifiable Rewards (PB-RLSVR), a novel framework that enhances multilingual reasoning by circumventing the need for human-annotated data in target languages. Our approach employs a high-performing English LLM as a "pivot" model to generate reference responses for reasoning tasks. A multilingual model is then rewarded based on the semantic equivalence of its responses to the English reference, effectively transferring the pivot model's reasoning capabilities across languages. We investigate several cross-lingual semantic reward functions, including those based on embeddings and machine translation. Extensive experiments on a suite of multilingual reasoning benchmarks show that our method significantly narrows the performance gap between English and other languages, substantially outperforming traditional PPO baselines. Specifically, our PB-RLSVR framework improves the average multilingual performance of Llama-3.1-8B-Instruct and Qwen3-32B by 16.41% and 10.17%, respectively, demonstrating a powerful and data-efficient approach to building truly multilingual reasoning agents.