# Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compression

arXiv: https://arxiv.org/abs/2405.01584 · HTML: https://mlsanity.com/paper/2405.01584/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 28 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1510.03820/) (2015)
- [Nonlinear Information Bottleneck](https://mlsanity.com/paper/1705.02436/) (2017)
- [Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2110.06530/) (2021)
- [A C-LSTM Neural Network for Text Classification](https://mlsanity.com/paper/1511.08630/) (2015)
- [Entropy and mutual information in models of deep neural networks](https://mlsanity.com/paper/1805.09785/) (2018)
- [Deep Dictionary Learning: A PARametric NETwork Approach](https://mlsanity.com/paper/1803.04022/) (2018)
- [When Dictionary Learning Meets Deep Learning: Deep Dictionary Learning and Coding Network for Image Recognition with Limited Data](https://mlsanity.com/paper/2005.10940/) (2020)
- [Dictionary Learning for Massive Matrix Factorization](https://mlsanity.com/paper/1605.00937/) (2016)
- [Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck](https://mlsanity.com/paper/1712.01272/) (2017)
- [Text Ranking and Classification using Data Compression](https://mlsanity.com/paper/2109.11577/) (2021)
- [Discriminative Bayesian Dictionary Learning for Classification](https://mlsanity.com/paper/1503.07989/) (2015)
- [A Sequential Neural Encoder with Latent Structured Description for Modeling Sentences](https://mlsanity.com/paper/1711.05433/) (2017)

## Built on by (most cited first, 1 of 1)

- [An Introduction to Compression-Based Machine Learning](https://mlsanity.com/paper/2609.21309/) (2026)

## Abstract (arXiv)

We propose a novel, lightweight supervised dictionary learning framework for text classification based on data compression and representation. This two-phase algorithm initially employs the Lempel-Ziv-Welch (LZW) algorithm to construct a dictionary from text datasets, focusing on the conceptual significance of dictionary elements. Subsequently, dictionaries are refined considering label data, optimizing dictionary atoms to enhance discriminative power based on mutual information and class distribution. This process generates discriminative numerical representations, facilitating the training of simple classifiers such as SVMs and neural networks. We evaluate our algorithm's information-theoretic performance using information bottleneck principles and introduce the information plane area rank (IPAR) as a novel metric to quantify the information-theoretic performance. Tested on six benchmark text datasets, our algorithm competes closely with top models, especially in limited-vocabulary contexts, using significantly fewer parameters. reviewOur algorithm closely matches top-performing models, deviating by only ~2% on limited-vocabulary datasets, using just 10% of their parameters. However, it falls short on diverse-vocabulary datasets, likely due to the LZW algorithm's constraints with low-repetition data. This contrast highlights its efficiency and limitations across different dataset types.