# SCALED : Surrogate-gradient for Codec-Aware Learning of Downsampling in ABR Streaming

arXiv: https://arxiv.org/abs/2602.00198 · HTML: https://mlsanity.com/paper/2602.00198/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Learned Image Downscaling for Upscaling using Content Adaptive Resampler](https://mlsanity.com/paper/1907.12904/) (2019)
- [Enhanced Standard Compatible Image Compression Framework based on Auxiliary Codec Networks](https://mlsanity.com/paper/2009.14754/) (2020)
- [Scale-arbitrary Invertible Image Downscaling](https://mlsanity.com/paper/2201.12576/) (2022)
- [Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers](https://mlsanity.com/paper/2402.05887/) (2024)
- [Efficient Evaluation of Quantization-Effects in Neural Codecs](https://mlsanity.com/paper/2502.04770/) (2025)
- [Video Compression with Arbitrary Rescaling Network](https://mlsanity.com/paper/2306.04202/) (2023)

## Built on by (most cited first, 1 of 1)

- [A Projection-Based Surrogate Gradient Interpretation for Neural Codec Wrappers](https://mlsanity.com/paper/2606.20671/) (2026)

## Abstract (arXiv)

The rapid growth in video consumption has introduced significant challenges to modern streaming architectures. Over-the-Top (OTT) video delivery now predominantly relies on Adaptive Bitrate (ABR) streaming, which dynamically adjusts bitrate and resolution based on client-side constraints such as display capabilities and network bandwidth. This pipeline typically involves downsampling the original high-resolution content, encoding and transmitting it, followed by decoding and upsampling on the client side. Traditionally, these processing stages have been optimized in isolation, leading to suboptimal end-to-end rate-distortion (R-D) performance. The advent of deep learning has spurred interest in jointly optimizing the ABR pipeline using learned resampling methods. However, training such systems end-to-end remains challenging due to the non-differentiable nature of standard video codecs, which obstructs gradient-based optimization. Recent works have addressed this issue using differentiable proxy models, based either on deep neural networks or hybrid coding schemes with differentiable components such as soft quantization, to approximate the codec behavior. While differentiable proxy codecs have enabled progress in compression-aware learning, they remain approximations that may not fully capture the behavior of standard, non-differentiable codecs. To our knowledge, there is no prior evidence demonstrating the inefficiencies of using standard codecs during training. In this work, we introduce a novel framework that enables end-to-end training with real, non-differentiable codecs by leveraging data-driven surrogate gradients derived from actual compression errors. It facilitates the alignment between training objectives and deployment performance. Experimental results show a 5.19% improvement in BD-BR (PSNR) compared to codec-agnostic training approaches, consistently across the entire rate-distortion convex hull spanning multiple downsampling ratios.