# TexShape: Information Theoretic Sentence Embedding for Language Models

arXiv: https://arxiv.org/abs/2402.05132 · HTML: https://mlsanity.com/paper/2402.05132/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 5 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations](https://mlsanity.com/paper/1707.00075/) (2017)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Improving Disentangled Text Representation Learning with Information-Theoretic Guidance](https://mlsanity.com/paper/2006.00693/) (2020)
- [CRYPTO-MINE: Cryptanalysis via Mutual Information Neural Estimation](https://mlsanity.com/paper/2309.08019/) (2023)

## Built on by (most cited first, 10 of 10)

- [Text summarization via global structure awareness](https://mlsanity.com/paper/2602.09821/) (2026)
- [Multi-level Reliability Interface for Semantic Communications over Wireless Networks](https://mlsanity.com/paper/2407.05487/) (2024)
- [PhonemeFake: Redefining Deepfake Realism with Language-Driven Segmental Manipulation and Adaptive Bilevel Detection](https://mlsanity.com/paper/2506.22783/) (2025)
- [SoK: Semantic Privacy in Large Language Models](https://mlsanity.com/paper/2506.23603/) (2025)
- [An Enhanced Text Compression Approach Using Transformer-based Language Models](https://mlsanity.com/paper/2412.15250/) (2024)
- [WavShape: Information-Theoretic Speech Representation Learning for Fair and Privacy-Aware Audio Processing](https://mlsanity.com/paper/2506.22789/) (2025)
- [MultiTok: Variable-Length Tokenization for Efficient LLMs Adapted from LZW Compression](https://mlsanity.com/paper/2410.21548/) (2024)
- [UniTAC: Universal Task-Aware Compression via Weighted Distortion Measures](https://mlsanity.com/paper/2608.16696/) (2026)
- [RoboShape: Information-Theoretic Point Cloud Representations for Privacy-Aware Robot Perception](https://mlsanity.com/paper/2608.21380/) (2026)
- [Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission](https://mlsanity.com/paper/2609.16279/) (2026)

## Abstract (arXiv)

With the exponential growth in data volume and the emergence of data-intensive applications, particularly in the field of machine learning, concerns related to resource utilization, privacy, and fairness have become paramount. This paper focuses on the textual domain of data and addresses challenges regarding encoding sentences to their optimized representations through the lens of information-theory. In particular, we use empirical estimates of mutual information, using the Donsker-Varadhan definition of Kullback-Leibler divergence. Our approach leverages this estimation to train an information-theoretic sentence embedding, called TexShape, for (task-based) data compression or for filtering out sensitive information, enhancing privacy and fairness. In this study, we employ a benchmark language model for initial text representation, complemented by neural networks for information-theoretic compression and mutual information estimations. Our experiments demonstrate significant advancements in preserving maximal targeted information and minimal sensitive information over adverse compression ratios, in terms of predictive accuracy of downstream models that are trained using the compressed data.