# Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Controlled Markovian System over a Communication Channel

arXiv: https://arxiv.org/abs/2411.13884 · HTML: https://mlsanity.com/paper/2411.13884/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 73 references that are in the map.
- First posted on arXiv 21 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 11 of 11)

- [Near Optimality of Finite Memory Feedback Policies in Partially Observed Markov Decision Processes](https://mlsanity.com/paper/2010.07452/) (2020)
- [Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces](https://mlsanity.com/paper/1503.02244/) (2015)
- [Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity](https://mlsanity.com/paper/2111.06781/) (2021)
- [Rate-cost tradeoffs in control](https://mlsanity.com/paper/1612.02126/) (2016)
- [LQG Control with Minimum Directed Information: Semidefinite Programming Approach](https://mlsanity.com/paper/1510.04214/) (2015)
- [Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments](https://mlsanity.com/paper/2311.00123/) (2023)
- [Semidefinite Programming Approach to Gaussian Sequential Rate-Distortion Trade-offs](https://mlsanity.com/paper/1411.7632/) (2014)
- [Zero-Delay Rate Distortion via Filtering for Vector-Valued Gaussian Sources](https://mlsanity.com/paper/1810.00298/) (2018)
- [Reinforcement Learning for Near-Optimal Design of Zero-Delay Codes for Markov Sources](https://mlsanity.com/paper/2311.12609/) (2023)
- [Time-invariant Prefix Coding for LQG Control](https://mlsanity.com/paper/2204.00588/) (2022)
- [Separated design of encoder and controller for networked linear quadratic optimal control](https://mlsanity.com/paper/1405.0135/) (2014)

## Abstract (arXiv)

We study the problem of joint optimization involving coding and control policies for a controlled Markovian sytem over a finite-rate noiseless communication channel. While structural results on the optimal encoding and control have been obtained in the literature, their implementation has been prohibitive in general, except for linear models. We develop regularity and existence results on optimal policies. We then obtain rigorous approximation and near optimality results for jointly optimal coding and control policies. To this end, we first develop existence, regularity, and structural properties on optimal policies, followed by rigorous approximations and reinforcement learning results. Notably, we establish near optimality of finite model approximations obtained via predictor quantization as well as sliding finite window approximations, and their reinforcement learning convergence to near optimality. A detailed comparison of the approximation schemes and their reinforcement learning performance is presented.