# Age-of-information minimization under energy harvesting and non-stationary environment

arXiv: https://arxiv.org/abs/2504.04916 · HTML: https://mlsanity.com/paper/2504.04916/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 23 of 23)

- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Whittle index based Q-learning for restless bandits with average reward](https://mlsanity.com/paper/2004.14427/) (2020)
- [Dynamic Regret of Policy Optimization in Non-stationary Environments](https://mlsanity.com/paper/2007.00148/) (2020)
- [A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitions](https://mlsanity.com/paper/1805.10066/) (2018)
- [Age of Information: Whittle Index for Scheduling Stochastic Arrivals](https://mlsanity.com/paper/1801.03422/) (2018)
- [A Whittle Index Approach to Minimizing Functions of Age of Information](https://mlsanity.com/paper/1908.10438/) (2019)
- [Q-Learning Lagrange Policies for Multi-Action Restless Bandits](https://mlsanity.com/paper/2106.12024/) (2021)
- [Optimistic Whittle Index Policy: Online Learning for Restless Bandits](https://mlsanity.com/paper/2205.15372/) (2022)
- [Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Control](https://mlsanity.com/paper/2010.03161/) (2020)
- [Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things](https://mlsanity.com/paper/1807.04356/) (2018)
- [A Reinforcement Learning Framework for Optimizing Age-of-Information in RF-powered Communication Systems](https://mlsanity.com/paper/1908.06367/) (2019)
- [AoI Minimization in Status Update Control with Energy Harvesting Sensors](https://mlsanity.com/paper/2009.04224/) (2020)
- [On learning Whittle index policy for restless bandits with scalable regret](https://mlsanity.com/paper/2202.03463/) (2022)
- [A Reinforcement Learning Approach to Age of Information in Multi-User Networks with HARQ](https://mlsanity.com/paper/2102.09774/) (2021)
- [An Online Learning Approach to Optimizing Time-Varying Costs of AoI](https://mlsanity.com/paper/2105.13383/) (2021)
- [Closed-Form Whittle's Index-Enabled Random Access for Timely Status Update](https://mlsanity.com/paper/1910.13871/) (2019)
- [On-Demand AoI Minimization in Resource-Constrained Cache-Enabled IoT Networks with Energy Harvesting Sensors](https://mlsanity.com/paper/2201.12277/) (2022)
- [Minimizing the AoI in Resource-Constrained Multi-Source Relaying Systems: Dynamic and Learning-based Scheduling](https://mlsanity.com/paper/2203.05656/) (2022)
- [On the Global Optimality of Whittle's index policy for minimizing the age of information](https://mlsanity.com/paper/2102.02528/) (2021)
- [Age-of-information minimization via opportunistic sampling by an energy harvesting source](https://mlsanity.com/paper/2201.02787/) (2022)
- [Learning to Minimize Age of Information over an Unreliable Channel with Energy Harvesting](https://mlsanity.com/paper/2106.16037/) (2021)
- [Whittle Index Based Scheduling Policy for Minimizing the Cost of Age of Information](https://mlsanity.com/paper/2109.05869/) (2021)
- [Whittle's index-based age-of-information minimization in multi-energy harvesting source networks](https://mlsanity.com/paper/2408.02570/) (2024)

## Abstract (arXiv)

This work focuses on minimizing the age of information for multiple energy harvesting sources that sample data and transmit it to a sink node. At each time, the central scheduler selects one of the sources to probe the quality of its channel to the sink node, and then the assessed channel quality is utilized to determine whether a source will sample and send the packet. For a single source case, we assume that the probed channel quality is known at each time instant, model the problem of AoI minimization as a Markov decision process, and prove the optimal sampling policy threshold structure. We then use this threshold structure and propose an AEC-SW-UCRL2 algorithm to handle unknown and time varying energy harvesting rate and channel statistics, motivated by the popular SWUCRL2 algorithm for non stationary reinforcement learning. This algorithm is applicable when an upper bound is available for the total variation of each of these quantities over a time horizon. Furthermore, in situations where these variation budgets are not accessible, we introduce the AEC-BORL algorithm, motivated by the well known BORL algorithm. For the multiple source case, we demonstrate that the AoI minimization problem can be formulated as a constrained MDP, which can be relaxed using a Lagrange multiplier and decoupled into sub problems across source nodes. We also derive Whittle index based source scheduling policy for probing and an optimal threshold policy for source sampling. We next leverage this Whittle index and threshold structure to develop the WIT-SW-UCRL2 algorithm for unknown time varying energy harvesting rates and channel statistics under their respective variation budgets. Moreover, we also proposed a Whittle index and threshold based bandit over reinforcement learning (WIT-BORL) algorithm for unknown variation budgets. Finally, we numerically demonstrate the efficacy of our algorithms.