TapWeight: Reweighting Pretraining Objectives for Task-Adaptive Pretraining
Not on the map. Why?
Abstract (arXiv)
Large-scale general domain pretraining followed by downstream-specific finetuning has become a predominant paradigm in machine learning. However, discrepancies between the pretraining and target domains can still lead to performance degradation in certain cases, underscoring the need for task-adaptive continued pretraining (TAP). TAP methods typically involve continued pretraining on task-specific unlabeled datasets or introducing additional unsupervised learning objectives to enhance model capabilities. While many TAP methods perform continued pretraining with multiple pretraining objectives, they often determine the tradeoff parameters between objectives manually, resulting in suboptimal outcomes and higher computational costs. In this paper, we propose TapWeight, a task-adaptive pretraining framework which automatically determines the optimal importance of each pretraining objective based on downstream feedback. TapWeight reweights each pretraining objective by solving a multi-level optimization problem. We applied TapWeight to both molecular property prediction and natural language understanding tasks, significantly surpassing baseline methods. Experimental results validate the effectiveness and generalizability of TapWeight.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 52 of its 60 references that are in the map. First posted on arXiv 13 Oct 2024.
Builds on
52 of 60 references.
References · 52
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
38 more
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
- 2019Strategies for Pre-training Graph Neural Networkscited by 971
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
- 2018End-to-End Multi-Task Learning with Attentioncited by 534
- 2019Meta-Learning with Implicit Gradientscited by 525
- 2016Cross-stitch Networks for Multi-task Learningcited by 467
- 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
- 2022UL2: Unifying Language Learning Paradigmscited by 295
- 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
- 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
- 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
- 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
- 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 126
- 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 124
- 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 105
- 2023DoGE: Domain Reweighting with Generalization Estimationcited by 90
- 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 73
- 2022Transcending Scaling Laws with 0.1% Extra Computecited by 57
- 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
- 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 34
- 2024AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learningcited by 31
- 2021Meta-Learning to Improve Pre-Trainingcited by 27
- 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
- 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 27
- 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
- 2023Making Scalable Meta Learning Practicalcited by 17
- 2021Domain-Adaptive Pretraining Methods for Dialogue Understandingcited by 11
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.