# Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling

arXiv: https://arxiv.org/abs/2507.14783 · HTML: https://mlsanity.com/paper/2507.14783/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition](https://mlsanity.com/paper/2310.05492/) (2023)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [General-Reasoner: Advancing LLM Reasoning Across All Domains](https://mlsanity.com/paper/2505.14652/) (2025)
- [MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline](https://mlsanity.com/paper/2401.08190/) (2024)
- [SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines](https://mlsanity.com/paper/2502.14739/) (2025)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [Magistral](https://mlsanity.com/paper/2506.10910/) (2025)
- [Command A: An Enterprise-Ready Large Language Model](https://mlsanity.com/paper/2504.00698/) (2025)
- [X-Reasoner: Towards Generalizable Reasoning Across Modalities and Domains](https://mlsanity.com/paper/2505.03981/) (2025)
- [Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data](https://mlsanity.com/paper/2409.12437/) (2024)

## Built on by (most cited first, 10 of 10)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Reinforcement Learning with Rubric Anchors](https://mlsanity.com/paper/2508.12790/) (2025)
- [Multi-Task GRPO: Reliable LLM Reasoning Across Tasks](https://mlsanity.com/paper/2602.05547/) (2026)
- [Advancing General-Purpose Reasoning Models with Modular Gradient Surgery](https://mlsanity.com/paper/2602.02301/) (2026)
- [Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR](https://mlsanity.com/paper/2606.25178/) (2026)
- [SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility](https://mlsanity.com/paper/2604.07837/) (2026)
- [Aligning Large Language Models with Searcher Preferences](https://mlsanity.com/paper/2603.10473/) (2026)
- [SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs](https://mlsanity.com/paper/2608.03573/) (2026)
- [Prompt-Level Reward Specifications for Open-Ended Post-Training](https://mlsanity.com/paper/2605.29275/) (2026)
- [LLM Agents for Automated Post-Training via Structured Recipe Optimization](https://mlsanity.com/paper/2610.18572/) (2026)

## Abstract (arXiv)

The pursuit of general-purpose artificial intelligence depends on large language models (LLMs) that can handle both structured reasoning and open-ended generation. We present Omni-Thinker, a unified reinforcement learning (RL) framework that scales LLMs across diverse tasks by combining hybrid rewards with backward-transfer-guided scheduling. Hybrid rewards integrate rule-based verifiable signals with preference-based evaluations from an LLM-as-a-Judge, enabling learning in both deterministic and subjective domains. Our scheduler orders tasks according to accuracy backward transfer (BWT), reducing forgetting and improving multi-task performance. Experiments across four domains show gains of 6.2% over joint training and 12.4% over model merging. Moreover, we demonstrate that simple assumptions on accuracy transfer yield accurate predictions of curriculum outcomes, with entropy dynamics explaining deviations due to generative tasks. These findings underscore the importance of BWT-aware scheduling and hybrid supervision for scaling RL-based post-training toward general-purpose LLMs.