# Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction

arXiv: https://arxiv.org/abs/1804.07849 · HTML: https://mlsanity.com/paper/1804.07849/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2018.

## Built on by (most cited first, 17 of 17)

- [An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels](https://mlsanity.com/paper/2203.11364/) (2022)
- [MINDE: Mutual Information Neural Diffusion Estimation](https://mlsanity.com/paper/2310.09031/) (2023)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality](https://mlsanity.com/paper/2211.15578/) (2022)
- [Information Theoretic Text-to-Image Alignment](https://mlsanity.com/paper/2405.20759/) (2024)
- [CLIP-Lite: Information Efficient Visual Representation Learning with Language Supervision](https://mlsanity.com/paper/2112.07133/) (2021)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [INFO-SEDD: Continuous Time Markov Chains as Scalable Information Metrics Estimators](https://mlsanity.com/paper/2502.19183/) (2025)
- [Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging](https://mlsanity.com/paper/2203.10315/) (2022)
- [MICO: Selective Search with Mutual Information Co-training](https://mlsanity.com/paper/2209.04378/) (2022)
- [Decomposed Mutual Information Optimization for Generalized Context in Meta-Reinforcement Learning](https://mlsanity.com/paper/2210.04209/) (2022)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [On planetary systems as ordered sequences](https://mlsanity.com/paper/2105.09966/) (2021)
- [Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?](https://mlsanity.com/paper/2206.14969/) (2022)
- [Review of Unsupervised POS Tagging and Its Implications on Language Acquisition](https://mlsanity.com/paper/2312.10169/) (2023)
- [ALICE: In-context, Zero-shot, Mutual Information Estimation](https://mlsanity.com/paper/2609.34962/) (2026)

## Abstract (arXiv)

We address part-of-speech (POS) induction by maximizing the mutual information between the induced label and its context. We focus on two training objectives that are amenable to stochastic gradient descent (SGD): a novel generalization of the classical Brown clustering objective and a recently proposed variational lower bound. While both objectives are subject to noise in gradient updates, we show through analysis and experiments that the variational lower bound is robust whereas the generalized Brown objective is vulnerable. We obtain competitive performance on a multitude of datasets and languages with a simple architecture that encodes morphology and context.