# Specializing Word Embeddings (for Parsing) by Information Bottleneck

arXiv: https://arxiv.org/abs/1910.00163 · HTML: https://mlsanity.com/paper/1910.00163/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 27 of 27)

- [Towards Modular LLMs by Building and Reusing a Library of LoRAs](https://mlsanity.com/paper/2405.11157/) (2024)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification](https://mlsanity.com/paper/2104.02862/) (2021)
- [Protecting Your LLMs with Information Bottleneck](https://mlsanity.com/paper/2404.13968/) (2024)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality](https://mlsanity.com/paper/2105.03484/) (2021)
- [Linguistic Dependencies and Statistical Dependence](https://mlsanity.com/paper/2104.08685/) (2021)
- [A Multi-Format Transfer Learning Model for Event Argument Extraction via Variational Information Bottleneck](https://mlsanity.com/paper/2208.13017/) (2022)
- [Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow](https://mlsanity.com/paper/2502.20750/) (2025)
- [VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck](https://mlsanity.com/paper/2601.05547/) (2026)
- [Hierarchical Interpretation of Neural Text Classification](https://mlsanity.com/paper/2202.09792/) (2022)
- [Rationalizing Predictions by Adversarial Information Calibration](https://mlsanity.com/paper/2301.06009/) (2023)
- [Expression Syntax Information Bottleneck for Math Word Problems](https://mlsanity.com/paper/2310.15664/) (2023)
- [Learning Distribution-Wise Control in Representation Space for Language Models](https://mlsanity.com/paper/2506.06686/) (2025)
- [Quantifying the Task-Specific Information in Text-Based Classifications](https://mlsanity.com/paper/2110.08931/) (2021)
- [Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality](https://mlsanity.com/paper/2301.13714/) (2023)
- [Span-based Named Entity Recognition by Generating and Compressing Information](https://mlsanity.com/paper/2302.05392/) (2023)
- [On Geodesic Distances and Contextual Embedding Compression for Text Classification](https://mlsanity.com/paper/2104.11295/) (2021)
- [Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction](https://mlsanity.com/paper/2308.08739/) (2023)
- [Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging](https://mlsanity.com/paper/2203.10315/) (2022)
- [Learning Intrinsic Dimension via Information Bottleneck for Explainable Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2402.18145/) (2024)
- [When does word order matter and when doesn't it?](https://mlsanity.com/paper/2402.18838/) (2024)
- [Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information](https://mlsanity.com/paper/2101.02879/) (2021)
- [Decoding Continuous Character-based Language from Non-invasive Brain Recordings](https://mlsanity.com/paper/2403.11183/) (2024)
- [Distilling Monolingual and Crosslingual Word-in-Context Representations](https://mlsanity.com/paper/2409.08719/) (2024)
- [Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction](https://mlsanity.com/paper/2607.11696/) (2026)

## Abstract (arXiv)

Pre-trained word embeddings like ELMo and BERT contain rich syntactic and semantic information, resulting in state-of-the-art performance on various tasks. We propose a very fast variational information bottleneck (VIB) method to nonlinearly compress these embeddings, keeping only the information that helps a discriminative parser. We compress each word embedding to either a discrete tag or a continuous vector. In the discrete version, our automatically compressed tags form an alternative tag set: we show experimentally that our tags capture most of the information in traditional POS tag annotations, but our tag sequences can be parsed more accurately at the same level of tag granularity. In the continuous version, we show experimentally that moderately compressing the word embeddings by our method yields a more accurate parser in 8 of 9 languages, unlike simple dimensionality reduction.