# A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers

arXiv: https://arxiv.org/abs/2606.20671 · HTML: https://mlsanity.com/paper/2606.20671/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Neural Video Compression with Feature Modulation](https://mlsanity.com/paper/2402.17414/) (2024)
- [Learned Image Downscaling for Upscaling using Content Adaptive Resampler](https://mlsanity.com/paper/1907.12904/) (2019)
- [Self-Conditioned Probabilistic Learning of Video Rescaling](https://mlsanity.com/paper/2107.11639/) (2021)
- [Enhanced Standard Compatible Image Compression Framework based on Auxiliary Codec Networks](https://mlsanity.com/paper/2009.14754/) (2020)
- [Scale-arbitrary Invertible Image Downscaling](https://mlsanity.com/paper/2201.12576/) (2022)
- [Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling](https://mlsanity.com/paper/2303.02353/) (2023)
- [Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers](https://mlsanity.com/paper/2402.05887/) (2024)
- [Learning a Virtual Codec Based on Deep Convolutional Neural Network to Compress Image](https://mlsanity.com/paper/1712.05969/) (2017)
- [Efficient Evaluation of Quantization-Effects in Neural Codecs](https://mlsanity.com/paper/2502.04770/) (2025)
- [Video Compression with Arbitrary Rescaling Network](https://mlsanity.com/paper/2306.04202/) (2023)
- [SCALED : Surrogate-gradient for Codec-Aware Learning of Downsampling in ABR Streaming](https://mlsanity.com/paper/2602.00198/) (2026)

## Abstract (arXiv)

Neural wrappers are learned pre-and postprocessing networks designed to enhance the performance of conventional video codecs. Although these approaches can significantly improve compression efficiency, training them remains challenging due to the non-differentiability of video codecs, which arises from the multiple discrete decisions involved in the encoding process. Surrogate gradients have recently emerged as an effective solution for enabling end-to-end learning with conventional codecs. They offer two main advantages: they avoid training an additional network to mimic the codec, and they can improve compression performance. In particular, the recently proposed SCALED method, which leverages the true compression error, has shown strong results for training neural pre-processors such as downscalers. However, this SCALED gradient was originally introduced as a reparameterization trick, which limits its interpretability. In this paper, we show that this surrogate gradient can be interpreted as a first-order local approximation of the video codec, providing insight into its effectiveness. We further demonstrate that it is effective not only for learning downscaling operations, but also for the more challenging task of full neural wrapping with pre-and post-processing networks. Finally, we show that the approach generalizes well across different video codecs, quality factors, and tasks, including multiple downscaling ratios, yielding BD-Rate (PSNR) reductions of up to -23.59% on x264 and -20.07% on VVenC relative to standard resampling baselines.