# Semantic-Aware Remote Estimation of Multiple Markov Sources Under Constraints

arXiv: https://arxiv.org/abs/2403.16855 · HTML: https://mlsanity.com/paper/2403.16855/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 21 of 21)

- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Semantics-Empowered Communication for Networked Intelligent Systems](https://mlsanity.com/paper/2007.11579/) (2020)
- [Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems](https://mlsanity.com/paper/1809.05149/) (2018)
- [A Critical Review of Communications in Multi-Robot Systems](https://mlsanity.com/paper/2206.09484/) (2022)
- [Age-of-Information vs. Value-of-Information Scheduling for Cellular Networked Control Systems](https://mlsanity.com/paper/1903.05356/) (2019)
- [Urgency of Information for Context-Aware Timely Status Updates in Remote Control Systems](https://mlsanity.com/paper/2002.07987/) (2020)
- [Model-Free Design of Control Systems over Wireless Fading Channels](https://mlsanity.com/paper/2009.01751/) (2020)
- [Control of Status Updates for Energy Harvesting Devices that Monitor Processes with Alarms](https://mlsanity.com/paper/1907.03826/) (2019)
- [Age of Actuation in a Wireless Power Transfer System](https://mlsanity.com/paper/2303.00507/) (2023)
- [Goal-oriented Policies for Cost of Actuation Error Minimization in Wireless Autonomous Systems](https://mlsanity.com/paper/2303.04908/) (2023)
- [Fundamental limits of remote estimation of autoregressive Markov processes under communication constraints](https://mlsanity.com/paper/1505.04829/) (2015)
- [Distortion Minimization with Age of Information and Cost Constraints](https://mlsanity.com/paper/2303.00850/) (2023)
- [Goal-oriented Estimation of Multiple Markov Sources in Resource-constrained Systems](https://mlsanity.com/paper/2311.07346/) (2023)
- [Optimal Scheduling for Discounted Age Penalty Minimization in Multi-Loop Networked Control](https://mlsanity.com/paper/1908.01503/) (2019)
- [Transmission Power Allocation for Remote Estimation with Multi-packet Reception Capabilities](https://mlsanity.com/paper/2101.12493/) (2021)
- [WiSwarm: Age-of-Information-based Wireless Networking for Collaborative Teams of UAVs](https://mlsanity.com/paper/2212.03298/) (2022)
- [Real-time Reconstruction of Markov Sources and Remote Actuation over Wireless Channels](https://mlsanity.com/paper/2302.13927/) (2023)
- [Remote Monitoring of Two-State Markov Sources via Random Access Channels: an Information Freshness vs. State Estimation Entropy Perspective](https://mlsanity.com/paper/2303.04507/) (2023)
- [Experimental Study of Transport Layer Protocols for Wireless Networked Control Systems](https://mlsanity.com/paper/2306.10578/) (2023)
- [State-aware Real-time Tracking and Remote Reconstruction of a Markov Source](https://mlsanity.com/paper/2309.11950/) (2023)

## Built on by (most cited first, 5 of 5)

- [Integrated Push-and-Pull Update Model for Goal-Oriented Effective Communication](https://mlsanity.com/paper/2407.14092/) (2024)
- [Hierarchy-Aware and Channel-Adaptive Semantic Communication for Bandwidth-Limited Data Fusion](https://mlsanity.com/paper/2503.17777/) (2025)
- [Feature-Based Semantics-Aware Scheduling for Energy-Harvesting Federated Learning](https://mlsanity.com/paper/2512.01983/) (2025)
- [Computing the Exact Pareto Front in Average-Cost Multi-Objective Markov Decision Processes](https://mlsanity.com/paper/2604.02196/) (2026)
- [Reinforcement Learning with Reward Machines for Sleep Control in Mobile Networks](https://mlsanity.com/paper/2604.07411/) (2026)

## Abstract (arXiv)

This paper studies the remote estimation of multiple Markov sources over a lossy and rate-constrained channel. Unlike most existing studies that treat all source states equally, we exploit the semantics of information and consider that the remote actuator has different tolerances for the estimation errors. We aim to find an optimal scheduling policy that minimizes the long-term textitstate-dependent costs of estimation errors under a transmission frequency constraint. The optimal scheduling problem is formulated as a constrained Markov decision process (CMDP). We show that the optimal Lagrangian cost follows a piece-wise linear and concave (PWLC) function, and the optimal policy is, at most, a randomized mixture of two simple deterministic policies. By exploiting the structural results, we develop a new textitintersection search algorithm that finds the optimal policy using only a few iterations. We further propose a reinforcement learning (RL) algorithm to compute the optimal policy without knowing textita priori the channel and source statistics. To avoid the ``curse of dimensionality" in MDPs, we propose an online low-complexity textitdrift-plus-penalty (DPP) algorithm. Numerical results show that continuous transmission is inefficient, and remarkably, our semantic-aware policies can attain the optimum by strategically utilizing fewer transmissions by exploiting the timing of the important information.