# Cache-Aware Reinforcement Learning in Large-Scale Recommender Systems

arXiv: https://arxiv.org/abs/2404.14961 · HTML: https://mlsanity.com/paper/2404.14961/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 11 of 11)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems](https://mlsanity.com/paper/2202.10842/) (2022)
- [Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems](https://mlsanity.com/paper/1902.05570/) (2019)
- [Deconfounding Duration Bias in Watch-time Prediction for Video Recommendation](https://mlsanity.com/paper/2206.06003/) (2022)
- [COLD: Towards the Next Generation of Pre-Ranking System](https://mlsanity.com/paper/2007.16122/) (2020)
- [Cascade Ranking for Operational E-commerce Search](https://mlsanity.com/paper/1706.02093/) (2017)
- [UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution](https://mlsanity.com/paper/2401.06470/) (2024)
- [Show me the Cache: Optimizing Cache-Friendly Recommendations for Sequential Content Access](https://mlsanity.com/paper/1805.06670/) (2018)

## Built on by (most cited first, 5 of 5)

- [Hierarchical LoRA MoE for Efficient CTR Model Scaling](https://mlsanity.com/paper/2510.10432/) (2025)
- [InterFormer: Effective Heterogeneous Interaction Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2411.09852/) (2024)
- [RPAF: A Reinforcement Prediction-Allocation Framework for Cache Allocation in Large-Scale Recommender Systems](https://mlsanity.com/paper/2409.13175/) (2024)
- [GRADE: Personalized Multi-Task Fusion via Group-relative Reinforcement Learning with Adaptive Dirichlet Exploration](https://mlsanity.com/paper/2510.07919/) (2025)
- [Memory Layer: Train the In-Model Cache for Recommendation Models](https://mlsanity.com/paper/2607.25110/) (2026)

## Abstract (arXiv)

Modern large-scale recommender systems are built upon computation-intensive infrastructure and usually suffer from a huge difference in traffic between peak and off-peak periods. In peak periods, it is challenging to perform real-time computation for each request due to the limited budget of computational resources. The recommendation with a cache is a solution to this problem, where a user-wise result cache is used to provide recommendations when the recommender system cannot afford a real-time computation. However, the cached recommendations are usually suboptimal compared to real-time computation, and it is challenging to determine the items in the cache for each user. In this paper, we provide a cache-aware reinforcement learning (CARL) method to jointly optimize the recommendation by real-time computation and by the cache. We formulate the problem as a Markov decision process with user states and a cache state, where the cache state represents whether the recommender system performs recommendations by real-time computation or by the cache. The computational load of the recommender system determines the cache state. We perform reinforcement learning based on such a model to improve user engagement over multiple requests. Moreover, we show that the cache will introduce a challenge called critic dependency, which deteriorates the performance of reinforcement learning. To tackle this challenge, we propose an eigenfunction learning (EL) method to learn independent critics for CARL. Experiments show that CARL can significantly improve the users' engagement when considering the result cache. CARL has been fully launched in Kwai app, serving over 100 million users.