# Crossword: A Semantic Approach to Data Compression via Masking

arXiv: https://arxiv.org/abs/2304.01106 · HTML: https://mlsanity.com/paper/2304.01106/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Semantic Communications: Principles and Challenges](https://mlsanity.com/paper/2201.01389/) (2021)
- [Task-Oriented Multi-User Semantic Communications for VQA Task](https://mlsanity.com/paper/2108.07357/) (2021)
- [Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation](https://mlsanity.com/paper/1905.05677/) (2019)
- [Robust Semantic Communications Against Semantic Noise](https://mlsanity.com/paper/2202.03338/) (2022)

## Built on by (most cited first, 1 of 1)

- [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024)

## Abstract (arXiv)

The traditional methods for data compression are typically based on the symbol-level statistics, with the information source modeled as a long sequence of i.i.d. random variables or a stochastic process, thus establishing the fundamental limit as entropy for lossless compression and as mutual information for lossy compression. However, the source (including text, music, and speech) in the real world is often statistically ill-defined because of its close connection to human perception, and thus the model-driven approach can be quite suboptimal. This study places careful emphasis on English text and exploits its semantic aspect to enhance the compression efficiency further. The main idea stems from the puzzle crossword, observing that the hidden words can still be precisely reconstructed so long as some key letters are provided. The proposed masking-based strategy resembles the above game. In a nutshell, the encoder evaluates the semantic importance of each word according to the semantic loss and then masks the minor ones, while the decoder aims to recover the masked words from the semantic context by means of the Transformer. Our experiments show that the proposed semantic approach can achieve much higher compression efficiency than the traditional methods such as Huffman code and UTF-8 code, while preserving the meaning in the target text to a great extent.