# An Empirical Analysis of Recurrent Learning Algorithms In Neural Lossy Image Compression Systems

arXiv: https://arxiv.org/abs/2201.11782 · HTML: https://mlsanity.com/paper/2201.11782/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 23 of 23)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks](https://mlsanity.com/paper/1703.10114/) (2017)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Towards Conceptual Compression](https://mlsanity.com/paper/1604.08772/) (2016)
- [Learning Image and Video Compression through Spatial-Temporal Energy Compaction](https://mlsanity.com/paper/1906.09683/) (2019)
- [Training recurrent networks online without backtracking](https://mlsanity.com/paper/1507.07680/) (2015)
- [The Neural State Pushdown Automata](https://mlsanity.com/paper/1909.05233/) (2019)
- [Learning to Improve Image Compression without Changing the Standard Decoder](https://mlsanity.com/paper/2009.12927/) (2020)
- [Learning Simpler Language Models with the Differential State Framework](https://mlsanity.com/paper/1703.08864/) (2017)
- [Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks](https://mlsanity.com/paper/1711.02326/) (2017)
- [Learned Neural Iterative Decoding for Lossy Image Compression Systems](https://mlsanity.com/paper/1803.05863/) (2018)
- [Sibling Neural Estimators: Improving Iterative Image Decoding with Gradient Communication](https://mlsanity.com/paper/1911.08478/) (2019)

## Built on by (most cited first, 2 of 2)

- [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023)
- [Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images](https://mlsanity.com/paper/2211.12047/) (2022)

## Abstract (arXiv)

Recent advances in deep learning have resulted in image compression algorithms that outperform JPEG and JPEG 2000 on the standard Kodak benchmark. However, they are slow to train (due to backprop-through-time) and, to the best of our knowledge, have not been systematically evaluated on a large variety of datasets. In this paper, we perform the first large-scale comparison of recent state-of-the-art hybrid neural compression algorithms, while exploring the effects of alternative training strategies (when applicable). The hybrid recurrent neural decoder is a former state-of-the-art model (recently overtaken by a Google model) that can be trained using backprop-through-time (BPTT) or with alternative algorithms like sparse attentive backtracking (SAB), unbiased online recurrent optimization (UORO), and real-time recurrent learning (RTRL). We compare these training alternatives along with the Google models (GOOG and E2E) on 6 benchmark datasets. Surprisingly, we found that the model trained with SAB performs better (outperforming even BPTT), resulting in faster convergence and a better peak signal-to-noise ratio.