mlsanityData to 7 Oct 2026

Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation

arXiv 2511.02463 ↗First posted 4 Nov 2025cs.AIcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards(RLVR) has demonstrated great potential in enhancing the reasoning capabilities of large language models (LLMs). However, its success has thus far been largely confined to the mathematical and programming domains with clear and automatically checkable outcomes. Reinforcement learning on open-ended tasks (e.g., creative writing and subjective Q&A) continues to rely on reward models due to the absence of verifiable solutions. This raises a key question: how can we extend RLVR to strengthen reasoning in open-ended tasks regardless of the absence of the unambiguous ground truth? To overcome this challenge, we introduce Verifiable Multiple-Choice Reformulation for Reinforcement Learning from Verifiable Rewards (VMR-RLVR), a novel training strategy that restructures open-ended data into verifiable multiple-choice formats, enabling effective training even in the absence of explicit ground truth. Experimental results on multiple benchmarks validate the effectiveness of our method in improving LLM performance on open-ended tasks. Notably, across seven open-ended benchmarks, our VMR-RLVR training delivers an average gain of 3.29 points over the RL with reward model.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 44 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

33 of 44 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Language Models that Think, Chat Better24 Sep
  2. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewards7 Aug
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  4. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers23 Jun
  5. 2025Reinforcing General Reasoning without Verifiers27 May
  6. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  10. 2025Process Reinforcement through Implicit Rewards3 Feb
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
3 more
  1. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  2. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  10. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  11. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  12. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
6 more
  1. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  2. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  3. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  4. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  5. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  6. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discoursecited by 2
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  3. 2026General Preference Reinforcement Learningcited by 1
  4. 2026OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  3. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  5. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  6. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains
  8. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation