mlsanityData to 7 Oct 2026

Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER

arXiv 2510.07566 ↗First posted 8 Oct 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that na"ive multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8% for NER and +8.8% for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 37 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

22 of 37 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  12. 2016Cross-stitch Networks for Multi-task Learningcited by 466
10 more
  1. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  2. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  3. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  4. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 102
  5. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  6. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  7. 2024NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Datacited by 34
  8. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  9. 2021NER-BERT: A Pre-trained Model for Low-Resource Entity Taggingcited by 21
  10. 2022Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysiscited by 6

Built on by

Not cited yet.