# Reinforcement Learning Based Approaches to Adaptive Context Caching in Distributed Context Management Systems

arXiv: https://arxiv.org/abs/2212.11709 · HTML: https://mlsanity.com/paper/2212.11709/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 44 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 7 of 7)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Context Aware Computing for The Internet of Things: A Survey](https://mlsanity.com/paper/1305.0982/) (2013)
- [A Reinforcement-Learning Approach to Proactive Caching in Wireless Networks](https://mlsanity.com/paper/1712.07084/) (2017)
- [The Role of Big Data Analytics in Industrial Internet of Things](https://mlsanity.com/paper/1904.05556/) (2019)
- [Time Aware Least Recent Used (TLRU) Cache Management Policy in ICN](https://mlsanity.com/paper/1801.00390/) (2018)
- [Learning-based Dynamic Cache Management in a Cloud](https://mlsanity.com/paper/1902.00795/) (2019)

## Abstract (arXiv)

Performance metrics-driven context caching has a profound impact on throughput and response time in distributed context management systems for real-time context queries. This paper proposes a reinforcement learning based approach to adaptively cache context with the objective of minimizing the cost incurred by context management systems in responding to context queries. Our novel algorithms enable context queries and sub-queries to reuse and repurpose cached context in an efficient manner. This approach is distinctive to traditional data caching approaches by three main features. First, we make selective context cache admissions using no prior knowledge of the context, or the context query load. Secondly, we develop and incorporate innovative heuristic models to calculate expected performance of caching an item when making the decisions. Thirdly, our strategy defines a time-aware continuous cache action space. We present two reinforcement learning agents, a value function estimating actor-critic agent and a policy search agent using deep deterministic policy gradient method. The paper also proposes adaptive policies such as eviction and cache memory scaling to complement our objective. Our method is evaluated using a synthetically generated load of context sub-queries and a synthetic data set inspired from real world data and query samples. We further investigate optimal adaptive caching configurations under different settings. This paper presents, compares, and discusses our findings that the proposed selective caching methods reach short- and long-term cost- and performance-efficiency. The paper demonstrates that the proposed methods outperform other modes of context management such as redirector mode, and database mode, and cache all policy by up to 60% in cost efficiency.