# Empower Sequence Labeling with Task-Aware Neural Language Model

arXiv: https://arxiv.org/abs/1709.04109 · HTML: https://mlsanity.com/paper/1709.04109/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2017.

## Built on by (most cited first, 28 of 28)

- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER](https://mlsanity.com/paper/2110.08454/) (2021)
- [A Span-Based Model for Joint Overlapped and Discontinuous Named Entity Recognition](https://mlsanity.com/paper/2106.14373/) (2021)
- [Self-Supervised Euphemism Detection and Identification for Content Moderation](https://mlsanity.com/paper/2103.16808/) (2021)
- [Structural Information Preserving for Graph-to-Text Generation](https://mlsanity.com/paper/2102.06749/) (2021)
- [Conformal prediction for text infilling and part-of-speech prediction](https://mlsanity.com/paper/2111.02592/) (2021)
- [Neural Structured Prediction for Inductive Node Classification](https://mlsanity.com/paper/2204.07524/) (2022)
- [TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained Tags](https://mlsanity.com/paper/2211.00684/) (2022)
- [Validating Label Consistency in NER Data Annotation](https://mlsanity.com/paper/2101.08698/) (2021)
- [kNN-NER: Named Entity Recognition with Nearest Neighbor Search](https://mlsanity.com/paper/2203.17103/) (2022)
- [Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies](https://mlsanity.com/paper/2402.09282/) (2024)
- [Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents](https://mlsanity.com/paper/2205.13434/) (2022)
- [GNN-SL: Sequence Labeling Based on Nearest Examples via GNN](https://mlsanity.com/paper/2212.02017/) (2022)
- [A Robust and Domain-Adaptive Approach for Low-Resource Named Entity Recognition](https://mlsanity.com/paper/2101.00388/) (2021)
- [#StayHome or #Marathon? Social Media Enhanced Pandemic Surveillance on Spatial-temporal Dynamic Graphs](https://mlsanity.com/paper/2108.03670/) (2021)
- [PSG: Prompt-based Sequence Generation for Acronym Extraction](https://mlsanity.com/paper/2111.14301/) (2021)
- [Named entity recognition architecture combining contextual and global features](https://mlsanity.com/paper/2112.08033/) (2021)
- [DMNER: Biomedical Entity Recognition by Detection and Matching](https://mlsanity.com/paper/2306.15736/) (2023)
- [CebuaNER: A New Baseline Cebuano Named Entity Recognition Model](https://mlsanity.com/paper/2310.00679/) (2023)
- [AutoTriggER: Label-Efficient and Robust Named Entity Recognition with Auxiliary Trigger Extraction](https://mlsanity.com/paper/2109.04726/) (2021)
- [Demonstration-based learning for few-shot biomedical named entity recognition under machine reading comprehension](https://mlsanity.com/paper/2308.06454/) (2023)
- [TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing](https://mlsanity.com/paper/2103.11441/) (2021)
- [Exploiting Global Contextual Information for Document-level Named Entity Recognition](https://mlsanity.com/paper/2106.00887/) (2021)
- [Using Domain Knowledge for Low Resource Named Entity Recognition](https://mlsanity.com/paper/2203.14738/) (2022)
- [Aspect and Opinion Term Extraction Using Graph Attention Network](https://mlsanity.com/paper/2404.19260/) (2024)
- [On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corpora](https://mlsanity.com/paper/2412.16976/) (2024)
- [Skill-based Explanations for Serendipitous Course Recommendation](https://mlsanity.com/paper/2508.19569/) (2025)

## Abstract (arXiv)

Linguistic sequence labeling is a general modeling approach that encompasses a variety of problems, such as part-of-speech tagging and named entity recognition. Recent advances in neural networks (NNs) make it possible to build reliable models without handcrafted features. However, in many cases, it is hard to obtain sufficient annotations to train these models. In this study, we develop a novel neural framework to extract abundant knowledge hidden in raw texts to empower the sequence labeling task. Besides word-level knowledge contained in pre-trained word embeddings, character-aware neural language models are incorporated to extract character-level knowledge. Transfer learning techniques are further adopted to mediate different components and guide the language model towards the key knowledge. Comparing to previous methods, these task-specific knowledge allows us to adopt a more concise model and conduct more efficient training. Different from most transfer learning methods, the proposed framework does not rely on any additional supervision. It extracts knowledge from self-contained order information of training sequences. Extensive experiments on benchmark datasets demonstrate the effectiveness of leveraging character-level knowledge and the efficiency of co-training. For example, on the CoNLL03 NER task, model training completes in about 6 hours on a single GPU, reaching F1 score of 91.71±0.10 without using any extra annotation.