# GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Exploration

arXiv: https://arxiv.org/abs/2510.07919 · HTML: https://mlsanity.com/paper/2510.07919/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 20 of 20)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS](https://mlsanity.com/paper/1912.06059/) (2019)
- [NPA: Neural News Recommendation with Personalized Attention](https://mlsanity.com/paper/1907.05559/) (2019)
- [Multi-Task Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2302.03525/) (2023)
- [QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou](https://mlsanity.com/paper/2411.11739/) (2024)
- [Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems](https://mlsanity.com/paper/2208.04560/) (2022)
- [Reinforcing User Retention in a Billion Scale Short Video Recommender System](https://mlsanity.com/paper/2302.01724/) (2023)
- [Value-aware Recommendation based on Reinforced Profit Maximization in E-commerce Systems](https://mlsanity.com/paper/1902.00851/) (2019)
- [UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution](https://mlsanity.com/paper/2401.06470/) (2024)
- [Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.14961/) (2024)
- [Ensure Timeliness and Accuracy: A Novel Sliding Window Data Stream Paradigm for Live Streaming Recommendation](https://mlsanity.com/paper/2402.14399/) (2024)
- [Uncertain Multi-Objective Recommendation via Orthogonal Meta-Learning Enhanced Bayesian Optimization](https://mlsanity.com/paper/2502.13180/) (2025)
- [UnifiedRL: A Reinforcement Learning Algorithm Tailored for Multi-Task Fusion in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.17589/) (2024)
- [A Vlogger-augmented Graph Neural Network Model for Micro-video Recommendation](https://mlsanity.com/paper/2405.18260/) (2024)

## Built on by (most cited first, 1 of 1)

- [DCEO: Direct Causal Effect Optimization for Long-Term User Value Modeling in E-commerce Search](https://mlsanity.com/paper/2608.25635/) (2026)

## Abstract (arXiv)

Balancing multiple objectives is critical for user satisfaction in modern recommender and search systems, yet current Multi-Task Fusion (MTF) methods rely on static, manually-tuned weights that fail to capture individual user intent. While Reinforcement Learning (RL) offers a path to personalization, traditional approaches often falter due to training instability and the sparse rewards inherent in these large-scale systems. To address these limitations, we propose Group-relative Reinforcement learning with Adaptive Dirichlet Exploration (GRADE), a novel and robust framework for personalized multi-task fusion. GRADE leverages a critic-free, Group Relative Policy Optimization (GRPO) paradigm, enabling stable and efficient policy learning by evaluating the relative performance of candidate weight groups. Its core innovations include employing the Dirichlet distribution for principled and structured exploration of the weight space, and a composite reward function that combines sparse user feedback with dense model priors and rule-based constraints to guide the search effectively. Deployed in the in-app marketplace of an application with over hundreds of millions daily active users, GRADE significantly outperforms established baselines, achieving substantial gains in rigorous large-scale A/B tests: +0.595% in CTR, +1.193% in CVR, +1.788% in OPM, and +1.568% in total order volume. Following its strong performance, GRADE has been fully deployed in the marketplace search scenario of Kuaishou, serving hundreds of millions of users.