# Small and Practical BERT Models for Sequence Labeling

arXiv: https://arxiv.org/abs/1909.00100 · HTML: https://mlsanity.com/paper/1909.00100/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2019.

## Built on by (most cited first, 30 of 40)

- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://mlsanity.com/paper/2101.03289/) (2021)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models](https://mlsanity.com/paper/2107.13686/) (2021)
- [Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages](https://mlsanity.com/paper/2109.10534/) (2021)
- [Intriguing Properties of Compression on Multilingual Models](https://mlsanity.com/paper/2211.02738/) (2022)
- [Multi-granular Adversarial Attacks against Black-box Neural Ranking Models](https://mlsanity.com/paper/2404.01574/) (2024)
- [MergeDistill: Merging Pre-trained Language Models using Distillation](https://mlsanity.com/paper/2106.02834/) (2021)
- [Distilling Large Language Models into Tiny and Effective Students using pQRNN](https://mlsanity.com/paper/2101.08890/) (2021)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks](https://mlsanity.com/paper/2103.15261/) (2021)
- [Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models](https://mlsanity.com/paper/2307.14134/) (2023)
- [AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models](https://mlsanity.com/paper/2201.08539/) (2022)
- [Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling](https://mlsanity.com/paper/2101.04758/) (2021)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)
- [Automatic Disfluency Detection from Untranscribed Speech](https://mlsanity.com/paper/2311.00867/) (2023)
- [LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation](https://mlsanity.com/paper/2103.06418/) (2021)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models](https://mlsanity.com/paper/2201.00558/) (2022)
- [Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learning](https://mlsanity.com/paper/2207.01012/) (2022)
- [RuSentEval: Linguistic Source, Encoder Force!](https://mlsanity.com/paper/2103.00573/) (2021)
- [Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Tagging](https://mlsanity.com/paper/2210.09840/) (2022)
- [Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?](https://mlsanity.com/paper/2304.11520/) (2023)
- [RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models](https://mlsanity.com/paper/2106.03613/) (2021)
- [Using Machine Translation to Localize Task Oriented NLG Output](https://mlsanity.com/paper/2107.04512/) (2021)
- [General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings](https://mlsanity.com/paper/2109.08449/) (2021)
- [Swing Distillation: A Privacy-Preserving Knowledge Distillation Framework](https://mlsanity.com/paper/2212.08349/) (2022)
- [Unsupervised Dense Retrieval Training with Web Anchors](https://mlsanity.com/paper/2305.05834/) (2023)

## Abstract (arXiv)

We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.