# Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

arXiv: https://arxiv.org/abs/2203.10315 · HTML: https://mlsanity.com/paper/2203.10315/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 19 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction](https://mlsanity.com/paper/2002.00737/) (2020)
- [Specializing Word Embeddings (for Parsing) by Information Bottleneck](https://mlsanity.com/paper/1910.00163/) (2019)
- [Unsupervised Parsing via Constituency Tests](https://mlsanity.com/paper/2010.03146/) (2020)
- [Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation](https://mlsanity.com/paper/1807.03121/) (2018)
- [Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles](https://mlsanity.com/paper/1612.06475/) (2016)
- [Unsupervised Neural Hidden Markov Models](https://mlsanity.com/paper/1609.09007/) (2016)
- [Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings](https://mlsanity.com/paper/1805.08237/) (2018)
- [Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction](https://mlsanity.com/paper/1804.07849/) (2018)
- [Unsupervised Learning of Syntactic Structure with Invertible Neural Projections](https://mlsanity.com/paper/1808.09111/) (2018)
- [Is POS Tagging Necessary or Even Helpful for Neural Dependency Parsing?](https://mlsanity.com/paper/2003.03204/) (2020)
- [Conditional Random Field Autoencoders for Unsupervised Structured Prediction](https://mlsanity.com/paper/1411.1147/) (2014)
- [Dependency Grammar Induction with Neural Lexicalization and Big Training Data](https://mlsanity.com/paper/1708.00801/) (2017)
- [CRF Autoencoder for Unsupervised Dependency Parsing](https://mlsanity.com/paper/1708.01018/) (2017)
- [Unsupervised POS Induction with Word Embeddings](https://mlsanity.com/paper/1503.06760/) (2015)

## Built on by (most cited first, 1 of 1)

- [Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages](https://mlsanity.com/paper/2506.17715/) (2025)

## Abstract (arXiv)

In recent years, large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, making it very challenging to incorporate contexualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tagging. The discriminative encoder of CRF-AE can straightforwardly incorporate ELMo word representations. Moreover, inspired by feature-rich HMM, we reintroduce hand-crafted features into the decoder of CRF-AE. Finally, experiments clearly show that our model outperforms previous state-of-the-art models by a large margin on Penn Treebank and multilingual Universal Dependencies treebank v2.0.