# RPAF: A Reinforcement Prediction-Allocation Framework for Cache Allocation in Large-Scale Recommender Systems

arXiv: https://arxiv.org/abs/2409.13175 · HTML: https://mlsanity.com/paper/2409.13175/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction](https://mlsanity.com/paper/2006.05639/) (2020)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems](https://mlsanity.com/paper/2202.10842/) (2022)
- [Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning](https://mlsanity.com/paper/1802.06501/) (2018)
- [COLD: Towards the Next Generation of Pre-Ranking System](https://mlsanity.com/paper/2007.16122/) (2020)
- [Large-scale Interactive Recommendation with Tree-structured Policy Gradient](https://mlsanity.com/paper/1811.05869/) (2018)
- [Cascade Ranking for Operational E-commerce Search](https://mlsanity.com/paper/1706.02093/) (2017)
- [Two-Stage Constrained Actor-Critic for Short Video Recommendation](https://mlsanity.com/paper/2302.01680/) (2023)
- [DCAF: A Dynamic Computation Allocation Framework for Online Serving System](https://mlsanity.com/paper/2006.09684/) (2020)
- [Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems](https://mlsanity.com/paper/2404.14961/) (2024)
- [RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation Allocation Approach for Recommender Systems](https://mlsanity.com/paper/2401.01369/) (2023)

## Built on by (most cited first, 1 of 1)

- [Adaptive2: Adaptive Domain Mining for Fine-grained Domain Adaptation Modeling](https://mlsanity.com/paper/2412.08198/) (2024)

## Abstract (arXiv)

Modern recommender systems are built upon computation-intensive infrastructure, and it is challenging to perform real-time computation for each request, especially in peak periods, due to the limited computational resources. Recommending by user-wise result caches is widely used when the system cannot afford a real-time recommendation. However, it is challenging to allocate real-time and cached recommendations to maximize the users' overall engagement. This paper shows two key challenges to cache allocation, i.e., the value-strategy dependency and the streaming allocation. Then, we propose a reinforcement prediction-allocation framework (RPAF) to address these issues. RPAF is a reinforcement-learning-based two-stage framework containing prediction and allocation stages. The prediction stage estimates the values of the cache choices considering the value-strategy dependency, and the allocation stage determines the cache choices for each individual request while satisfying the global budget constraint. We show that the challenge of training RPAF includes globality and the strictness of budget constraints, and a relaxed local allocator (RLA) is proposed to address this issue. Moreover, a PoolRank algorithm is used in the allocation stage to deal with the streaming allocation problem. Experiments show that RPAF significantly improves users' engagement under computational budget constraints.