# Gibbs randomness-compression proposition

arXiv: https://arxiv.org/abs/2505.23869 · HTML: https://mlsanity.com/paper/2505.23869/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 58 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Deep Compressed Sensing](https://mlsanity.com/paper/1905.06723/) (2019)
- [Why Random Pruning Is All We Need to Start Sparse](https://mlsanity.com/paper/2210.02412/) (2022)
- [Compression-aware Training of Neural Networks using Frank-Wolfe](https://mlsanity.com/paper/2205.11921/) (2022)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Sparsification via Compressed Sensing for Automatic Speech Recognition](https://mlsanity.com/paper/2102.04932/) (2021)

## Abstract (arXiv)

A proposition that connects randomness and compression is put forward via Gibbs entropy over set of measurement vectors associated with a lossy compression process. In building this connection, we use a performance of a learning task as a probe of compression in iterative compress-train cycles. This can be thought as iterative coarse-graining from statistical mechanics perspective using thermodynamic efficiency as a probe. We formulate this connection via comonotonic relationship within a very small decrease in compression ratio and the performance. We have showcase the validity of this proposition with a canonical vision task in deep learning with three different model compression processes as it a baseline model. We use the following, simpler to more complex model compression approaches: (1) random pruning,(2) magnitude pruning, and (3) a more complex compression by using dual tomographic compression, which utilizes compressed sensing in dual fashion which is introduced as a new method. We use remaining weights of deep learning network as a measurement vector where we measure the Gibbs entropy. We show case the idea that there is an inherent computable connection between compression probed by performance and randomness from an entropy measure on the learned model.