mlsanityData to 14 Oct 2026

Ensembling Finetuned Language Models for Text Classification

arXiv 2410.19889 ↗First posted 25 Oct 2024cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Finetuning is a common practice widespread across different communities to adapt pretrained models to particular tasks. Text classification is one of these tasks for which many pretrained models are available. On the other hand, ensembles of neural networks are typically used to boost performance and provide reliable uncertainty estimates. However, ensembling pretrained models for text classification is not a well-studied avenue. In this paper, we present a metadataset with predictions from five large finetuned models on six datasets, and report results of different ensembling strategies from these predictions. Our results shed light on how ensembling can improve the performance of finetuned text classifiers and incentivize future adoption of ensembles in such tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

24 of 39 references.

References · 24

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  12. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
12 more
  1. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  2. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  3. 2022Efficient Few-Shot Learning Without Promptscited by 137
  4. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  5. 2023Generative AI Text Classification using Ensemble LLM Approachescited by 29
  6. 2017The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representationscited by 23
  7. 2023Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shiftcited by 20
  8. 2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 18
  9. 2023Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenMLcited by 9
  10. 2022Multi-Objective Model Selection for Time Series Forecastingcited by 8
  11. 2023CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failurecited by 8
  12. 2023Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoMLcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 2
  2. 2026Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicitycited by 0