# Pragmatic Image Compression for Human-in-the-Loop Decision-Making arXiv: https://arxiv.org/abs/2108.04219 ยท HTML: https://mlsanity.com/paper/2108.04219/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 52 references that are in the map. - First posted on arXiv 7 Jul 2021. ## Builds on (most cited first, 30 of 36) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013) - [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014) - [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019) - [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020) - [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014) - [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016) - [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016) - [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015) - [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016) - [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018) - [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020) - [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016) - [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021) - [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020) - [Designing an Encoder for StyleGAN Image Manipulation](https://mlsanity.com/paper/2102.02766/) (2021) - [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020) - [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018) - [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017) - [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016) - [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015) - [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018) - [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017) - [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018) - [Practical Lossless Compression with Latent Variables using Bits Back Coding](https://mlsanity.com/paper/1901.04866/) (2019) - [Learning Convolutional Networks for Content-weighted Image Compression](https://mlsanity.com/paper/1703.10553/) (2017) - [Predicting Video with VQVAE](https://mlsanity.com/paper/2103.01950/) (2021) ## Built on by (most cited first, 7 of 7) - [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022) - [Control-Aware Prediction Objectives for Autonomous Driving](https://mlsanity.com/paper/2204.13319/) (2022) - [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023) - [Pragmatic Communication in Multi-Agent Collaborative Perception](https://mlsanity.com/paper/2401.12694/) (2024) - [Perceptual Group Tokenizer: Building Perception with Iterative Grouping](https://mlsanity.com/paper/2311.18296/) (2023) - [Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces](https://mlsanity.com/paper/2111.14951/) (2021) - [On Task Scope and Information Retention in Source Coding](https://mlsanity.com/paper/2609.37575/) (2026) ## Abstract (arXiv) Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by a user for downstream tasks -- e.g., deciding which product to click on in a shopping website -- is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's decisions. We approach this problem by training a compression model through human-in-the-loop learning as the user performs tasks with the compressed images. The key insight is to train the model to produce a compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, we train a discriminator that tries to distinguish whether a user's action was taken in response to the compressed image or the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing a car racing video game. The results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compression model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.