mlsanityData to 14 Oct 2026

Lossy Compression for Lossless Prediction

arXiv 2106.10800 ↗First posted 21 Jun 2021cs.LG cs.IT math.ITcited by 52

Not on the map. Why?

Abstract (arXiv)

Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than 1000× on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 64 of its 169 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

64 of 169 references.

References · 64

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  12. 2020Supervised Contrastive Learningcited by 2,829
38 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  2. 2017Deep Setscited by 1,421
  3. 2016Group Equivariant Convolutional Networkscited by 1,165
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2018Variational image compression with a scale hyperpriorcited by 935
  6. 2016Deep Variational Information Bottleneckcited by 930
  7. 2016End-to-end Optimized Image Compressioncited by 696
  8. 2018Joint Autoregressive and Hierarchical Priors for Learned Image Compressioncited by 664
  9. 2019On Variational Bounds of Mutual Informationcited by 559
  10. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  11. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  12. 2020High-Fidelity Generative Image Compressioncited by 338
  13. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  14. 2017Lossy Image Compression with Compressive Autoencoderscited by 294
  15. 2020CompressAI: a PyTorch library and evaluation platform for end-to-end compression researchcited by 276
  16. 2020Channel-wise Autoregressive Entropy Models for Learned Image Compressioncited by 264
  17. 2018Generative Adversarial Networks for Extreme Learned Image Compressioncited by 219
  18. 2019Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoffcited by 211
  19. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  20. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  21. 2016Variational Lossy Autoencodercited by 176
  22. 2020A Theory of Usable Information Under Computational Constraintscited by 162
  23. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  24. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  25. 2018Context-adaptive Entropy Model for End-to-end Optimized Image Compressioncited by 143
  26. 2020Nonlinear Transform Codingcited by 136
  27. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  28. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  29. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  30. 2017Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networkscited by 92
  31. 2020Improving Inference for Neural Image Compressioncited by 83
  32. 2020On the Benefits of Invariance in Neural Networkscited by 79
  33. 2020The Conditional Entropy Bottleneckcited by 75
  34. 2020Universally Quantized Neural Compressioncited by 64
  35. 2020End-to-end Learning of Compressible Featurescited by 43
  36. 2015Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Networkcited by 39
  37. 2020Compressing Images by Encoding Their Latent Representations with Relative Entropy Codingcited by 39
  38. 2020Minimum Excess Risk in Bayesian Learningcited by 35

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 8; 2024: 9; 2025: 8; 2026: 9.

52 papers.

Sort:

By area:

  1. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
  2. 2023Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Modelscited by 94
  3. 2022An Introduction to Neural Data Compressioncited by 79
  4. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  5. 2021Task-Oriented Communication for Multi-Device Cooperative Edge Inferencecited by 52
  6. 2021Optimal Representations for Covariate Shiftcited by 51
  7. 2022Information-theoretic Online Memory Selection for Continual Learningcited by 42
  8. 2021Compressive Visual Representationscited by 41
  9. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  10. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  11. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 34
  12. 2021Supervised Compression for Resource-Constrained Edge Computing Systemscited by 31
38 more
  1. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  2. 2022Augment with Care: Contrastive Learning for Combinatorial Problemscited by 22
  3. 2022SC2 Benchmark: Supervised Compression for Split Computingcited by 15
  4. 2022What Do We Maximize in Self-Supervised Learning?cited by 15
  5. 2022Task-Oriented Communication for Edge Video Analyticscited by 13
  6. 2023FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computingcited by 12
  7. 2024SMC++: Masked Learning of Unsupervised Video Semantic Compressioncited by 12
  8. 2024Pragmatic Communication in Multi-Agent Collaborative Perceptioncited by 10
  9. 2024A Generative Model of Symmetry Transformationscited by 10
  10. 2025Lossy Neural Compression for Geospatial Analytics: A Reviewcited by 10
  11. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  12. 2022Compressed Vision for Efficient Video Understandingcited by 8
  13. 2022Perceptual Video Coding for Machines via Satisfied Machine Ratio Modelingcited by 7
  14. 2025NeuCo-Bench: A Novel Benchmark Framework for Neural Embeddings in Earth Observationcited by 7
  15. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  16. 2023Task-aware Distributed Source Coding under Dynamic Bandwidthcited by 6
  17. 2024Task-Aware Encoder Control for Deep Video Compressioncited by 6
  18. 2023Towards understanding neural collapse in supervised contrastive learning with the information bottleneck methodcited by 5
  19. 2022When less is more: Simplifying inputs aids neural network understandingcited by 4
  20. 2025RIS-aided Latent Space Alignment for Semantic Channel Equalizationcited by 2
  21. 2025An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentationcited by 2
  22. 2022Task-Aware Network Coding Over Butterfly Networkcited by 1
  23. 2024Neural Embedding Compression For Efficient Multi-Task Earth Observation Modellingcited by 1
  24. 2024Adversarial Robustness of Bottleneck Injected Deep Neural Networks for Task-Oriented Communicationcited by 1
  25. 2025Aligning Task- and Reconstruction-Oriented Communications for Edge Intelligencecited by 1
  26. 2025Image Coding for Machines via Feature-Preserving Rate-Distortion Optimizationcited by 1
  27. 2025Rate-Distortion Optimized Communication for Collaborative Perceptioncited by 1
  28. 2026Lossy Common Information in a Learnable Gray-Wyner Networkcited by 1
  29. 2026TextBoost: Boosting Scene Text Fidelity in Ultra-low Bitrate Image Compressioncited by 1
  30. 2026See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RLcited by 1
  31. 2023Ideal Abstractions for Decision-Focused Learningcited by 0
  32. 2024Residual-INR: Communication Efficient On-Device Learning Using Implicit Neural Representationcited by 0
  33. 2024Efficient Compression of Sparse Accelerator Data Using Implicit Neural Representations and Importance Samplingcited by 0
  34. 2025Task-Oriented Co-Design of Communication, Computing, and Control for Edge-Enabled Industrial Cyber-Physical Systemscited by 0
  35. 2026GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compressioncited by 0
  36. 2026AdaRadar: Rate Adaptive Spectral Compression for Radar-based Perceptioncited by 0
  37. 2026Hadamard-Domain Model Quantization for Learned Image Codingcited by 0
  38. 2026How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KLcited by 0

50 most cited of 52 All 52 (JSON)