# Differentiable JPEG: The Devil is in the Details

arXiv: https://arxiv.org/abs/2309.06978 · HTML: https://mlsanity.com/paper/2309.06978/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [HiDDeN: Hiding Data With Deep Networks](https://mlsanity.com/paper/1807.09937/) (2018)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [Invertible Image Signal Processing](https://mlsanity.com/paper/2103.15061/) (2021)
- [The Rate-Distortion-Accuracy Tradeoff: JPEG Case Study](https://mlsanity.com/paper/2008.00605/) (2020)
- [FaceGuard: Proactive Deepfake Detection](https://mlsanity.com/paper/2109.05673/) (2021)
- [Learning to Improve Image Compression without Changing the Standard Decoder](https://mlsanity.com/paper/2009.12927/) (2020)
- [Differentiable Data Augmentation with Kornia](https://mlsanity.com/paper/2011.09832/) (2020)
- [Towards Robust Data Hiding Against (JPEG) Compression: A Pseudo-Differentiable Deep Learning Approach](https://mlsanity.com/paper/2101.00973/) (2020)

## Built on by (most cited first, 13 of 13)

- [Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast](https://mlsanity.com/paper/2402.08567/) (2024)
- [GuardSplat: Efficient and Robust Watermarking for 3D Gaussian Splatting](https://mlsanity.com/paper/2411.19895/) (2024)
- [Transferable Adversarial Attacks on Black-Box Vision-Language Models](https://mlsanity.com/paper/2505.01050/) (2025)
- [MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Models](https://mlsanity.com/paper/2510.01549/) (2025)
- [Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics](https://mlsanity.com/paper/2408.01541/) (2024)
- [PnP-CM: Consistency Models as Plug-and-Play Priors for Inverse Problems](https://mlsanity.com/paper/2509.22736/) (2025)
- [VMAF Re-implementation on PyTorch: Some Experimental Results](https://mlsanity.com/paper/2310.15578/) (2023)
- [Big Brother is Watching: Proactive Deepfake Detection via Learnable Hidden Face](https://mlsanity.com/paper/2504.11309/) (2025)
- [A Perspective on Deep Vision Performance with Standard Image and Video Codecs](https://mlsanity.com/paper/2404.12330/) (2024)
- [On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling](https://mlsanity.com/paper/2506.21874/) (2025)
- [FAST-DIPS: Adjoint-Free Analytic Steps and Hard-Constrained Likelihood Correction for Diffusion-Prior Inverse Problems](https://mlsanity.com/paper/2603.01591/) (2026)
- [MetaCloak-JPEG: JPEG-Robust Adversarial Perturbation for Preventing Unauthorized DreamBooth-Based Deepfake Generation](https://mlsanity.com/paper/2604.18537/) (2026)
- [Versatile Recompression-Aware Perceptual Image Super-Resolution](https://mlsanity.com/paper/2511.18090/) (2025)

## Abstract (arXiv)

JPEG remains one of the most widespread lossy image coding methods. However, the non-differentiable nature of JPEG restricts the application in deep learning pipelines. Several differentiable approximations of JPEG have recently been proposed to address this issue. This paper conducts a comprehensive review of existing diff. JPEG approaches and identifies critical details that have been missed by previous methods. To this end, we propose a novel diff. JPEG approach, overcoming previous limitations. Our approach is differentiable w.r.t. the input image, the JPEG quality, the quantization tables, and the color conversion parameters. We evaluate the forward and backward performance of our diff. JPEG approach against existing methods. Additionally, extensive ablations are performed to evaluate crucial design choices. Our proposed diff. JPEG resembles the (non-diff.) reference implementation best, significantly surpassing the recent-best diff. approach by 3.47dB (PSNR) on average. For strong compression rates, we can even improve PSNR by 9.51dB. Strong adversarial attack results are yielded by our diff. JPEG, demonstrating the effective gradient approximation. Our code is available at https://github.com/necla-ml/Diff-JPEG.