mlsanityData to 7 Oct 2026

Small and Practical BERT Models for Sequence Labeling

arXiv 1909.00100 ↗First posted 31 Aug 2019cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 7; 2023: 6; 2024: 5; 2025: 1; 2026: 1.

40 papers.

Sort:

By area:

  1. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  2. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  3. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  4. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  5. 2024A Survey on Transformer Compressioncited by 44
  6. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  7. 2021Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languagescited by 24
  8. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  9. 2024Multi-granular Adversarial Attacks against Black-box Neural Ranking Modelscited by 14
  10. 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
  11. 2021Distilling Large Language Models into Tiny and Effective Students using pQRNNcited by 10
  12. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
28 more
  1. 2021One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networkscited by 8
  2. 2023Developing and Evaluating Tiny to Medium-Sized Turkish BERT Modelscited by 8
  3. 2022AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Modelscited by 7
  4. 2021Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labelingcited by 6
  5. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Modelscited by 6
  6. 2023Automatic Disfluency Detection from Untranscribed Speechcited by 6
  7. 2021LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillationcited by 5
  8. 2021Morph Call: Probing Morphosyntactic Content of Multilingual Transformerscited by 5
  9. 2022Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Modelscited by 5
  10. 2022Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learningcited by 5
  11. 2021RuSentEval: Linguistic Source, Encoder Force!cited by 4
  12. 2022Graph-Based Multilingual Label Propagation for Low-Resource Part-of-Speech Taggingcited by 4
  13. 2023Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?cited by 2
  14. 2021RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Modelscited by 1
  15. 2021Using Machine Translation to Localize Task Oriented NLG Outputcited by 1
  16. 2021General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddingscited by 1
  17. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  18. 2023Unsupervised Dense Retrieval Training with Web Anchorscited by 1
  19. 2023Co-training and Co-distillation for Quality Improvement and Compression of Language Modelscited by 1
  20. 2021BERT-based knowledge extraction method of unstructured domain textcited by 0
  21. 2021Zero-Resource Multi-Dialectal Arabic Natural Language Understandingcited by 0
  22. 2021MOROCCO: Model Resource Comparison Frameworkcited by 0
  23. 2021Why Can You Lay Off Heads? Investigating How BERT Heads Transfercited by 0
  24. 2021Classification-based Quality Estimation: Small and Efficient Models for Real-world Applicationscited by 0
  25. 2023E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizercited by 0
  26. 2024TAMS: Translation-Assisted Morphological Segmentationcited by 0
  27. 2025Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Taskscited by 0
  28. 2026A Lightweight Hybrid Transformer-CRF Architecture for Multi-Type Bangla Medical Entity Recognitioncited by 0