mlsanityData to 13 Oct 2026

Cluster & Tune: Boost Cold Start Performance in Text Classification

arXiv 2203.10581 ↗First posted 20 Mar 2022cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In real-world scenarios, a text classification task often begins with a cold start, when labeled data is scarce. In such cases, the common practice of fine-tuning pre-trained models, such as BERT, for a target classification task, is prone to produce poor performance. We suggest a method to boost the performance of such models by adding an intermediate unsupervised classification task, between the pre-training and fine-tuning phases. As such an intermediate task, we perform clustering and train the pre-trained model on predicting the cluster labels. We test this hypothesis on various data sets, and show that this additional classification phase can significantly improve performance, mainly for topical classification tasks, when the number of labeled instances available for fine-tuning is only a couple of dozen to a few hundred.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

37 of 70 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,011
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,779
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  9. 2016Matching Networks for One Shot Learningcited by 2,251
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,942
  11. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,590
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,556
25 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,023
  2. 2019Passage Re-ranking with BERTcited by 638
  3. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  4. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  5. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  6. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  7. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  8. 2019Not Enough Data? Deep Learning to the Rescue!cited by 175
  9. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  10. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  11. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 70
  12. 2019Induction Networks for Few-Shot Text Classificationcited by 64
  13. 2019Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classificationcited by 62
  14. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  15. 2020From Arguments to Key Points: Towards Automatic Argument Summarizationcited by 60
  16. 2019Corpus Wide Argument Mining -- a Working Solutioncited by 36
  17. 2020Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillationcited by 25
  18. 2015LSHTC: A Benchmark for Large-Scale Text Classificationcited by 22
  19. 2020Adaptive Self-training for Few-shot Neural Sequence Labelingcited by 18
  20. 2020To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscited by 10
  21. 2019An Effective Domain Adaptive Post-Training Method for BERT in Response Selectioncited by 7
  22. 2016Semi-supervised Clustering for Short Text via Deep Representation Learningcited by 5
  23. 2020Pre-training Text Representations as Meta Learningcited by 5
  24. 2019To Tune or Not To Tune? How About the Best of Both Worlds?cited by 4
  25. 2019Unsupervised Transfer Learning via BERT Neuron Selectioncited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 3; 2026: 2.

13 papers.

Sort:

  1. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  2. 2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
  3. 2024Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Bettercited by 11
  4. 2022Will It Blend? Mixing Training Paradigms & Prompting for Argument Quality Predictioncited by 7
  5. 2023QAID: Question Answering Inspired Few-shot Intent Detectioncited by 5
  6. 2023Towards Understanding the Effect of Pretraining Label Granularitycited by 3
  7. 2023Analyzing Text Representations by Measuring Task Alignmentcited by 2
  8. 2025DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learningcited by 2
  9. 2024Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Studycited by 1
  10. 2025Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languagescited by 1
  11. 2025Few-Shot Query Intent Detection via Relation-Aware Prompt Learningcited by 1
  12. 2026Breaking the Generator Barrier: Disentangled Representation for Generalizable AI-Text Detectioncited by 1
1 more
  1. 2026BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialectscited by 0