# Compressed-Language Models for Understanding Compressed File Formats: a JPEG Exploration

arXiv: https://arxiv.org/abs/2405.17146 · HTML: https://mlsanity.com/paper/2405.17146/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers](https://mlsanity.com/paper/2305.07185/) (2023)
- [SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check](https://mlsanity.com/paper/2004.14166/) (2020)
- [Beyond Language Models: Byte Models are Digital World Simulators](https://mlsanity.com/paper/2402.19155/) (2024)
- [RGB no more: Minimally-decoded JPEG Vision Transformers](https://mlsanity.com/paper/2211.16421/) (2022)
- [Bytes Are All You Need: Transformers Operating Directly On File Bytes](https://mlsanity.com/paper/2306.00238/) (2023)
- [Training LLMs over Neurally Compressed Text](https://mlsanity.com/paper/2404.03626/) (2024)
- [DCT-domain Deep Convolutional Neural Networks for Multiple JPEG Compression Classification](https://mlsanity.com/paper/1712.02313/) (2017)

## Built on by (most cited first, 4 of 4)

- [LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing](https://mlsanity.com/paper/2406.07714/) (2024)
- [JPEG-LM: LLMs as Image Generators with Canonical Codec Representations](https://mlsanity.com/paper/2408.08459/) (2024)
- [Proxy Compression for Language Modeling](https://mlsanity.com/paper/2602.04289/) (2026)
- [Transformers from Compressed Representations](https://mlsanity.com/paper/2510.23665/) (2025)

## Abstract (arXiv)

This study investigates whether Compressed-Language Models (CLMs), i.e. language models operating on raw byte streams from Compressed File Formats~(CFFs), can understand files compressed by CFFs. We focus on the JPEG format as a representative CFF, given its commonality and its representativeness of key concepts in compression, such as entropy coding and run-length encoding. We test if CLMs understand the JPEG format by probing their capabilities to perform along three axes: recognition of inherent file properties, handling of files with anomalies, and generation of new files. Our findings demonstrate that CLMs can effectively perform these tasks. These results suggest that CLMs can understand the semantics of compressed data when directly operating on the byte streams of files produced by CFFs. The possibility to directly operate on raw compressed files offers the promise to leverage some of their remarkable characteristics, such as their ubiquity, compactness, multi-modality and segment-nature.