# Bit Allocation using Optimization

arXiv: https://arxiv.org/abs/2209.09422 · HTML: https://mlsanity.com/paper/2209.09422/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 77 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Variable Rate Deep Image Compression With a Conditional Autoencoder](https://mlsanity.com/paper/1909.04802/) (2019)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [Practical Full Resolution Learned Lossless Image Compression](https://mlsanity.com/paper/1811.12817/) (2018)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [Content Adaptive Optimization for Neural Image Compression](https://mlsanity.com/paper/1906.01223/) (2019)
- [Overfitting for Fun and Profit: Instance-Adaptive Data Compression](https://mlsanity.com/paper/2101.08687/) (2021)
- [Hierarchical Autoregressive Modeling for Neural Video Compression](https://mlsanity.com/paper/2010.10258/) (2020)
- [End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression](https://mlsanity.com/paper/2112.09529/) (2021)
- [Flexible Neural Image Compression via Code Editing](https://mlsanity.com/paper/2209.09244/) (2022)
- [Flexible-Rate Learned Hierarchical Bi-Directional Video Compression With Motion Refinement and Frame-Level Bit Allocation](https://mlsanity.com/paper/2206.13613/) (2022)

## Built on by (most cited first, 9 of 9)

- [NVRC: Neural Video Representation Compression](https://mlsanity.com/paper/2409.07414/) (2024)
- [Offline and Online Optical Flow Enhancement for Deep Video Compression](https://mlsanity.com/paper/2307.05092/) (2023)
- [Large Language Model for Lossless Image Compression with Visual Prompts](https://mlsanity.com/paper/2502.16163/) (2025)
- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [Robustly overfitting latents for flexible neural image compression](https://mlsanity.com/paper/2401.17789/) (2024)
- [Learned Rate Control for Frame-Level Adaptive Neural Video Compression via Dynamic Neural Network](https://mlsanity.com/paper/2508.20709/) (2025)
- [Correcting the Sub-optimal Bit Allocation](https://mlsanity.com/paper/2209.14575/) (2022)
- [Adaptive Rate Control for Deep Video Compression with Rate-Distortion Prediction](https://mlsanity.com/paper/2412.18834/) (2024)

## Abstract (arXiv)

In this paper, we consider the problem of bit allocation in Neural Video Compression (NVC). First, we reveal a fundamental relationship between bit allocation in NVC and Semi-Amortized Variational Inference (SAVI). Specifically, we show that SAVI with GoP (Group-of-Picture)-level likelihood is equivalent to pixel-level bit allocation with precise rate & quality dependency model. Based on this equivalence, we establish a new paradigm of bit allocation using SAVI. Different from previous bit allocation methods, our approach requires no empirical model and is thus optimal. Moreover, as the original SAVI using gradient ascent only applies to single-level latent, we extend the SAVI to multi-level such as NVC by recursively applying back-propagating through gradient ascent. Finally, we propose a tractable approximation for practical implementation. Our method can be applied to scenarios where performance outweights encoding speed, and serves as an empirical bound on the R-D performance of bit allocation. Experimental results show that current state-of-the-art bit allocation algorithms still have a room of ≈ 0.5 dB PSNR to improve compared with ours. Code is available at urlhttps://github.com/tongdaxu/Bit-Allocation-Using-Optimization.