# Semi-supervised sequence tagging with bidirectional language models

arXiv: https://arxiv.org/abs/1705.00108 · HTML: https://mlsanity.com/paper/1705.00108/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2017.

## Built on by (most cited first, 30 of 86)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [SituatedQA: Incorporating Extra-Linguistic Contexts into QA](https://mlsanity.com/paper/2109.06157/) (2021)
- [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024)
- [AR-LSAT: Investigating Analytical Reasoning of Text](https://mlsanity.com/paper/2104.06598/) (2021)
- [Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense](https://mlsanity.com/paper/2202.05749/) (2022)
- [From LSAT: The Progress and Challenges of Complex Reasoning](https://mlsanity.com/paper/2108.00648/) (2021)
- [CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learning](https://mlsanity.com/paper/2212.10341/) (2022)
- [Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences](https://mlsanity.com/paper/2301.08571/) (2023)
- [Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation](https://mlsanity.com/paper/2212.08632/) (2022)
- [Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative](https://mlsanity.com/paper/2109.07437/) (2021)
- [Position: Measure Dataset Diversity, Don't Just Claim It](https://mlsanity.com/paper/2407.08188/) (2024)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [There is a Time and Place for Reasoning Beyond the Image](https://mlsanity.com/paper/2203.00758/) (2022)
- [Decomposed Meta-Learning for Few-Shot Named Entity Recognition](https://mlsanity.com/paper/2204.05751/) (2022)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [DCT-Former: Efficient Self-Attention with Discrete Cosine Transform](https://mlsanity.com/paper/2203.01178/) (2022)
- [FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue](https://mlsanity.com/paper/2205.06262/) (2022)
- [Reasoning over Hybrid Chain for Table-and-Text Open Domain QA](https://mlsanity.com/paper/2201.05880/) (2022)
- [Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey](https://mlsanity.com/paper/2304.11461/) (2023)
- [Recent Trends in Named Entity Recognition (NER)](https://mlsanity.com/paper/2101.11420/) (2021)
- [Computer Science Named Entity Recognition in the Open Research Knowledge Graph](https://mlsanity.com/paper/2203.14579/) (2022)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [Ensemble Learning for Large Language Models in Text and Code Generation: A Survey](https://mlsanity.com/paper/2503.13505/) (2025)
- [SpanProto: A Two-stage Span-based Prototypical Network for Few-shot Named Entity Recognition](https://mlsanity.com/paper/2210.09049/) (2022)
- [Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews](https://mlsanity.com/paper/2104.05861/) (2021)
- [Implicit Temporal Reasoning for Evidence-Based Fact-Checking](https://mlsanity.com/paper/2302.12569/) (2023)
- [Several categories of Large Language Models (LLMs): A Short Survey](https://mlsanity.com/paper/2307.10188/) (2023)
- [Masked Conditional Random Fields for Sequence Labeling](https://mlsanity.com/paper/2103.10682/) (2021)
- [A Prototypical Semantic Decoupling Method via Joint Contrastive Learning for Few-Shot Name Entity Recognition](https://mlsanity.com/paper/2302.13610/) (2023)

## Abstract (arXiv)

Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pre- trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.