# A Dual-Critic Reinforcement Learning Framework for Frame-level Bit Allocation in HEVC/H.265

arXiv: https://arxiv.org/abs/2104.01735 · HTML: https://mlsanity.com/paper/2104.01735/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 9 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 2 of 2)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Reinforced Bit Allocation under Task-Driven Semantic Distortion Metrics](https://mlsanity.com/paper/1910.07392/) (2019)

## Built on by (most cited first, 5 of 5)

- [MuZero with Self-competition for Rate Control in VP9 Video Compression](https://mlsanity.com/paper/2202.06626/) (2022)
- [RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression](https://mlsanity.com/paper/2501.12216/) (2025)
- [Neural Frank-Wolfe Policy Optimization for Region-of-Interest Intra-Frame Coding with HEVC/H.265](https://mlsanity.com/paper/2209.13210/) (2022)
- [IQNet: Image Quality Assessment Guided Just Noticeable Difference Prefiltering For Versatile Video Coding](https://mlsanity.com/paper/2312.09799/) (2023)
- [Action-Constrained Reinforcement Learning for Frame-Level Bit Allocation in HEVC/H.265 through Frank-Wolfe Policy Optimization](https://mlsanity.com/paper/2203.05127/) (2022)

## Abstract (arXiv)

This paper introduces a dual-critic reinforcement learning (RL) framework to address the problem of frame-level bit allocation in HEVC/H.265. The objective is to minimize the distortion of a group of pictures (GOP) under a rate constraint. Previous RL-based methods tackle such a constrained optimization problem by maximizing a single reward function that often combines a distortion and a rate reward. However, the way how these rewards are combined is usually ad hoc and may not generalize well to various coding conditions and video sequences. To overcome this issue, we adapt the deep deterministic policy gradient (DDPG) reinforcement learning algorithm for use with two critics, with one learning to predict the distortion reward and the other the rate reward. In particular, the distortion critic works to update the agent when the rate constraint is satisfied. By contrast, the rate critic makes the rate constraint a priority when the agent goes over the bit budget. Experimental results on commonly used datasets show that our method outperforms the bit allocation scheme in x265 and the single-critic baseline by a significant margin in terms of rate-distortion performance while offering fairly precise rate control.