# UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution

arXiv: https://arxiv.org/abs/2401.06470 · HTML: https://mlsanity.com/paper/2401.06470/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2024.

## Built on by (most cited first, 5 of 5)

- [Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.14961/) (2024)
- [GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Exploration](https://mlsanity.com/paper/2510.07919/) (2025)
- [Supervised Learning-enhanced Multi-Group Actor Critic for Live Stream Allocation in Feed](https://mlsanity.com/paper/2412.10381/) (2024)
- [A Practical Introduction to Deep Reinforcement Learning](https://mlsanity.com/paper/2505.08295/) (2025)
- [MaRCA: Multi-Agent Reinforcement Learning for Dynamic Computation Allocation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2512.24325/) (2025)

## Abstract (arXiv)

In recent years, there has been a growing interest in utilizing reinforcement learning (RL) to optimize long-term rewards in recommender systems. Since industrial recommender systems are typically designed as multi-stage systems, RL methods with a single agent face challenges when optimizing multiple stages simultaneously. The reason is that different stages have different observation spaces, and thus cannot be modeled by a single agent. To address this issue, we propose a novel UNidirectional-EXecution-based multi-agent Reinforcement Learning (UNEX-RL) framework to reinforce the long-term rewards in multi-stage recommender systems. We show that the unidirectional execution is a key feature of multi-stage recommender systems, bringing new challenges to the applications of multi-agent reinforcement learning (MARL), namely the observation dependency and the cascading effect. To tackle these challenges, we provide a cascading information chain (CIC) method to separate the independent observations from action-dependent observations and use CIC to train UNEX-RL effectively. We also discuss practical variance reduction techniques for UNEX-RL. Finally, we show the effectiveness of UNEX-RL on both public datasets and an online recommender system with over 100 million users. Specifically, UNEX-RL reveals a 0.558% increase in users' usage time compared with single-agent RL algorithms in online A/B experiments, highlighting the effectiveness of UNEX-RL in industrial recommender systems.