# Learning Image and Video Compression through Spatial-Temporal Energy Compaction

arXiv: https://arxiv.org/abs/1906.09683 · HTML: https://mlsanity.com/paper/1906.09683/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2019.

## Built on by (most cited first, 27 of 27)

- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [Implicit Neural Video Compression](https://mlsanity.com/paper/2112.11312/) (2021)
- [Perceptual Learned Video Compression with Recurrent Conditional GAN](https://mlsanity.com/paper/2109.03082/) (2021)
- [Extending Neural P-frame Codecs for B-frame Coding](https://mlsanity.com/paper/2104.00531/) (2021)
- [Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression](https://mlsanity.com/paper/2401.15864/) (2024)
- [MobileCodec: Neural Inter-frame Video Compression on Mobile Devices](https://mlsanity.com/paper/2207.08338/) (2022)
- [Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set](https://mlsanity.com/paper/2111.10302/) (2021)
- [Advancing Learned Video Compression with In-loop Frame Prediction](https://mlsanity.com/paper/2211.07004/) (2022)
- [Hierarchical B-frame Video Coding Using Two-Layer CANF without Motion Coding](https://mlsanity.com/paper/2304.02690/) (2023)
- [LC-FDNet: Learned Lossless Image Compression with Frequency Decomposition Network](https://mlsanity.com/paper/2112.06417/) (2021)
- [IPDAE: Improved Patch-Based Deep Autoencoder for Lossy Point Cloud Geometry Compression](https://mlsanity.com/paper/2208.02519/) (2022)
- [Estimating the Resize Parameter in End-to-end Learned Image Compression](https://mlsanity.com/paper/2204.12022/) (2022)
- [Prediction and Reference Quality Adaptation for Learned Video Compression](https://mlsanity.com/paper/2406.14118/) (2024)
- [End-to-End Learned Image Compression with Quantized Weights and Activations](https://mlsanity.com/paper/2111.09348/) (2021)
- [You Can Mask More For Extremely Low-Bitrate Image Compression](https://mlsanity.com/paper/2306.15561/) (2023)
- [NVC-1B: A Large Neural Video Coding Model](https://mlsanity.com/paper/2407.19402/) (2024)
- [An Empirical Analysis of Recurrent Learning Algorithms In Neural Lossy Image Compression Systems](https://mlsanity.com/paper/2201.11782/) (2022)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [CSIM: A Copula-based similarity index sensitive to local changes for Image quality assessment](https://mlsanity.com/paper/2410.01411/) (2024)
- [Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding](https://mlsanity.com/paper/2506.07709/) (2025)
- [CADC: Content Adaptive Diffusion-Based Generative Image Compression](https://mlsanity.com/paper/2602.21591/) (2026)
- [Neural JPEG: End-to-End Image Compression Leveraging a Standard JPEG Encoder-Decoder](https://mlsanity.com/paper/2201.11795/) (2022)
- [Transformations in Learned Image Compression from a Modulation Perspective](https://mlsanity.com/paper/2203.02158/) (2022)
- [Foveation-based Deep Video Compression without Motion Search](https://mlsanity.com/paper/2203.16490/) (2022)

## Abstract (arXiv)

Compression has been an important research topic for many decades, to produce a significant impact on data transmission and storage. Recent advances have shown a great potential of learning image and video compression. Inspired from related works, in this paper, we present an image compression architecture using a convolutional autoencoder, and then generalize image compression to video compression, by adding an interpolation loop into both encoder and decoder sides. Our basic idea is to realize spatial-temporal energy compaction in learning image and video compression. Thereby, we propose to add a spatial energy compaction-based penalty into loss function, to achieve higher image compression performance. Furthermore, based on temporal energy distribution, we propose to select the number of frames in one interpolation loop, adapting to the motion characteristics of video contents. Experimental results demonstrate that our proposed image compression outperforms the latest image compression standard with MS-SSIM quality metric, and provides higher performance compared with state-of-the-art learning compression methods at high bit rates, which benefits from our spatial energy compaction approach. Meanwhile, our proposed video compression approach with temporal energy compaction can significantly outperform MPEG-4 and is competitive with commonly used H.264. Both our image and video compression can produce more visually pleasant results than traditional standards.