# Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-time Popularities

arXiv: https://arxiv.org/abs/1708.06698 · HTML: https://mlsanity.com/paper/1708.06698/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2017.

## Built on by (most cited first, 15 of 15)

- [Ensemble Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.02618/) (2023)
- [Distributed Reinforcement Learning for Privacy-Preserving Dynamic Edge Caching](https://mlsanity.com/paper/2110.10349/) (2021)
- [Optimistic No-regret Algorithms for Discrete Caching](https://mlsanity.com/paper/2208.06414/) (2022)
- [Whittle Index based Q-Learning for Wireless Edge Caching with Linear Function Approximation](https://mlsanity.com/paper/2202.13187/) (2022)
- [Digital Twin-Assisted Data-Driven Optimization for Reliable Edge Caching in Wireless Networks](https://mlsanity.com/paper/2407.00286/) (2024)
- [Online Caching with Optimistic Learning](https://mlsanity.com/paper/2202.10590/) (2022)
- [Cocktail Edge Caching: Ride Dynamic Trends of Content Popularity with Ensemble Learning](https://mlsanity.com/paper/2101.05885/) (2021)
- [Hybrid Policy Learning for Energy-Latency Tradeoff in MEC-Assisted VR Video Service](https://mlsanity.com/paper/2104.01036/) (2021)
- [Online Caching with no Regret: Optimistic Learning via Recommendations](https://mlsanity.com/paper/2204.09345/) (2022)
- [Unsupervised Recurrent Federated Learning for Edge Popularity Prediction in Privacy-Preserving Mobile Edge Computing Networks](https://mlsanity.com/paper/2207.00755/) (2022)
- [Predictive Edge Caching through Deep Mining of Sequential Patterns in User Content Retrievals](https://mlsanity.com/paper/2210.02657/) (2022)
- [Demystifying Reinforcement Learning in Time-Varying Systems](https://mlsanity.com/paper/2201.05560/) (2022)
- [NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation](https://mlsanity.com/paper/2411.04138/) (2024)
- [Reinforcement Learning for Physical Layer Communications](https://mlsanity.com/paper/2106.11595/) (2021)
- [Caching Contents with Varying Popularity using Restless Bandits](https://mlsanity.com/paper/2212.03291/) (2022)

## Abstract (arXiv)

Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this work, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.