mlsanityData to 7 Oct 2026

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

arXiv 1811.01088 ↗First posted 2 Nov 2018cs.CLcited by 214

Not on the map. Why?

Abstract (arXiv)

Pretraining sentence encoders with language modeling and related unsupervised tasks has recently been shown to be very effective for language understanding tasks. By supplementing language model-style pretraining with further training on data-rich supervised tasks, such as natural language inference, we obtain additional performance improvements on the GLUE benchmark. Applying supplementary training on BERT (Devlin et al., 2018), we attain a GLUE score of 81.8---the state of the art (as of 02/24/2019) and a 1.4 point improvement over BERT. We also observe reduced variance across random restarts in this setting. Our approach yields similar improvements when applied to ELMo (Peters et al., 2018a) and Radford et al. (2018)'s model. In addition, the benefits of supplementary training are particularly pronounced in data-constrained regimes, as we show in experiments with artificially limited training data.

Key facts. Cited by 214 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 60; 2022: 70; 2023: 49; 2024: 20; 2025: 9; 2026: 6.

214 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  4. 2021True Few-Shot Learning with Language Modelscited by 343
  5. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  6. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  7. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  8. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  9. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
38 more
  1. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  2. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  3. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2021Entailment as Few-Shot Learnercited by 122
  6. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  7. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  8. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  9. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  10. 2022Fusing finetuned models for better pretrainingcited by 110
  11. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  12. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  13. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  14. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  15. 2023Deep Model Fusion: A Surveycited by 89
  16. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  17. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  18. 2021How to Train BERT with an Academic Budgetcited by 80
  19. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  20. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  21. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  22. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  23. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  24. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  25. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  26. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  27. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  28. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  29. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  30. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  31. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  32. 2022Memory Efficient Continual Learning with Transformerscited by 43
  33. 2022IDPG: An Instance-Dependent Prompt Generation Methodcited by 43
  34. 2022Adaptive Transformers for Robust Few-shot Cross-domain Face Anti-spoofingcited by 38
  35. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  36. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  37. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  38. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35

50 most cited of 214 All 214 (JSON)