# Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Control

arXiv: https://arxiv.org/abs/2601.18069 · HTML: https://mlsanity.com/paper/2601.18069/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion for compression and wireless, one of 14 directions in Diffusion models.
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Placed in

- Diffusion models / [Diffusion for compression and wireless](https://mlsanity.com/diffusion/diffusion-for-compression-and-wireless/)
- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Applications of Deep Reinforcement Learning in Communications and Networking: A Survey](https://mlsanity.com/paper/1810.07862/) (2018)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [Age of Information: An Introduction and Survey](https://mlsanity.com/paper/2007.08564/) (2020)
- [Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimization](https://mlsanity.com/paper/2308.05384/) (2023)
- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services](https://mlsanity.com/paper/2303.13052/) (2023)
- [Urgency of Information for Context-Aware Timely Status Updates in Remote Control Systems](https://mlsanity.com/paper/2002.07987/) (2020)
- [The Age of Gossip in Networks](https://mlsanity.com/paper/2102.02893/) (2021)
- [Risk-Aware Optimization of Age of Information in the Internet of Things](https://mlsanity.com/paper/2002.09805/) (2020)
- [Version age-based client scheduling policy for federated learning](https://mlsanity.com/paper/2402.05407/) (2024)
- [Version Age of Information in Clustered Gossip Networks](https://mlsanity.com/paper/2109.08669/) (2021)
- [Version Age-Optimal Cached Status Updates in a Gossiping Network with Energy Harvesting Sensor](https://mlsanity.com/paper/2311.01909/) (2023)
- [A Deep Reinforcement Learning Approach for Improving Age of Information in Mission-Critical IoT](https://mlsanity.com/paper/2311.13861/) (2023)
- [Statistical Age of Information: A Risk-Aware Metric and Its Applications in Status Updates](https://mlsanity.com/paper/2406.02139/) (2024)
- [Offline and Distributional Reinforcement Learning for Wireless Communications](https://mlsanity.com/paper/2504.03804/) (2025)
- [Goal-oriented Tensor: Beyond Age of Information Towards Semantics-Empowered Goal-Oriented Communications](https://mlsanity.com/paper/2307.00535/) (2023)
- [Version Innovation Age and Age of Incorrect Version for Monitoring Markovian Sources](https://mlsanity.com/paper/2401.17691/) (2024)
- [Optimizing Version AoI in Energy-Harvesting IoT: Model-Based and Learning-Based Approaches](https://mlsanity.com/paper/2510.00904/) (2025)

## Built on by (most cited first, 1 of 1)

- [From Denoising to Decision Making: A Survey on Diffusion Model-Enabled Deep Reinforcement Learning for Wireless Networks](https://mlsanity.com/paper/2605.25531/) (2026)

## Abstract (arXiv)

Ensuring timely and semantically accurate information delivery is critical in real-time wireless systems. While Age of Information (AoI) quantifies temporal freshness, Version Age of Information (VAoI) captures semantic staleness by accounting for version evolution between transmitters and receivers. Existing VAoI scheduling approaches primarily focus on minimizing average VAoI, overlooking rare but severe staleness events that can compromise reliability under stochastic packet arrivals and unreliable channels. This paper investigates both average-oriented and tail-risk-sensitive VAoI scheduling in a multi-user status update system with long-term transmission cost constraints. We first formulate the average VAoI minimization problem as a constrained Markov decision process and introduce a deep diffusion-based Soft Actor-Critic (D2SAC) algorithm. By generating actions through a diffusion-based denoising process, D2SAC enhances policy expressiveness and establishes a strong baseline for mean performance. Building on this foundation, we put forth RS-D3SAC, a risk-sensitive deep distributional diffusion-based Soft Actor-Critic algorithm. RS-D3SAC integrates a diffusion-based actor with a quantile-based distributional critic, explicitly modeling the full VAoI return distribution. This enables principled tail-risk optimization via Conditional Value-at-Risk (CVaR) while satisfying long-term transmission cost constraints. Extensive simulations show that, while D2SAC reduces average VAoI, RS-D3SAC consistently achieves substantial reductions in CVaR without sacrificing mean performance. The dominant gain in tail-risk reduction stems from the distributional critic, with the diffusion-based actor providing complementary refinement to stabilize and enrich policy decisions, highlighting their effectiveness for robust and risk-aware VAoI scheduling in multi-user wireless systems.