mlsanityData to 7 Oct 2026

Diversity-Enhanced Reasoning for Subjective Questions

arXiv 2507.20187 ↗First posted 27 Jul 2025cs.CLcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Reasoning Models (LRMs) with long chain-of-thought capabilities, optimized via reinforcement learning with verifiable rewards (RLVR), excel at objective reasoning tasks like mathematical problem solving and code generation. However, RLVR is known for degrading generation diversity, which causes LRMs to fall short on subjective reasoning that has multiple answers depending on different role perspectives. While recent studies recognize the importance of diversity-enhanced training in objective reasoning, limited attention has been given to subjective tasks. In this paper, we find that subjective reasoning can be improved by introducing perspective diversity and token-level diversity, with the former one providing a coherent scaffolding anchored to a real-world stakeholder group and the latter one broadening the answer search space. We propose MultiRole-R1, a diversity-enhanced training framework featuring an unsupervised data construction pipeline that synthesizes reasoning chains incorporating various role perspectives. It also employs reinforcement learning via Group Relative Policy Optimization with reward shaping, taking diversity as a reward signal in addition to verifiable reward. Training on subjective tasks solely, MultiRole-R1 increases the in-domain and out-of-domain accuracy by 14.1% and 7.64%, and even enhances the performance on advanced math reasoning such as AIME 2024. We further show that diversity is a more consistent indicator of accuracy than reasoning length.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 76 of its 101 references that are in the map. First posted on arXiv 27 Jul 2025.

Builds on

76 of 101 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  6. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 69

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
38 more
  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  3. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  6. 2025LIMO: Less is More for Reasoningcited by 488
  7. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  8. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  10. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  11. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  12. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  13. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  14. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  15. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  16. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  17. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  18. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  19. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  20. 2025S*: Test Time Scaling for Code Generationcited by 95
  21. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  22. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  23. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  24. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  25. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  26. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  27. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  28. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  29. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  30. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  31. 2025Scaling Laws of Synthetic Data for Language Modelscited by 44
  32. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  33. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  34. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  35. 2025Hogwild! Inference: Parallel LLM Generation via Concurrent Attentioncited by 32
  36. 2023Diversity of Thought Improves Reasoning Abilities of LLMscited by 24
  37. 2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
  38. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  2. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 10
  3. 2025Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learningcited by 5
  4. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 5
  5. 2025Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertaintycited by 3
  6. 2026Code2Math: Can Your Code Agent Effectively Evolve Math Problems Through Exploration?cited by 2
  7. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  8. 2026Large Language Models Explore by Latent Distillingcited by 0
  9. 2026Evaluating the Hidden Costs of Personalization in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning
  2. 2025Diversity-Incentivized Exploration for Versatile Reasoning
  3. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
  4. 2026Diverse Thinking Schemata Elicit Better Reasoning in Large Language Models
  5. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
  6. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
  7. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  8. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning