# Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models

arXiv: https://arxiv.org/abs/2412.11455 · HTML: https://mlsanity.com/paper/2412.11455/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [BioRED: A Rich Biomedical Relation Extraction Dataset](https://mlsanity.com/paper/2204.04263/) (2022)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Identifying beneficial task relations for multi-task learning in deep neural networks](https://mlsanity.com/paper/1702.08303/) (2017)
- [Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions](https://mlsanity.com/paper/2305.10435/) (2023)
- [Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation](https://mlsanity.com/paper/2106.09017/) (2021)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning](https://mlsanity.com/paper/1904.04153/) (2019)
- [RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements](https://mlsanity.com/paper/2411.15700/) (2024)

## Built on by (most cited first, 6 of 6)

- [MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning](https://mlsanity.com/paper/2502.15954/) (2025)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [Retrieval-augmented in-context learning for multimodal large language models in disease classification](https://mlsanity.com/paper/2505.02087/) (2025)
- [Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation](https://mlsanity.com/paper/2509.04534/) (2025)
- [EPEE: Towards Efficient and Effective Foundation Models in Biomedicine](https://mlsanity.com/paper/2503.02053/) (2025)
- [TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning](https://mlsanity.com/paper/2605.31025/) (2026)

## Abstract (arXiv)

To efficiently select optimal dataset combinations for enhancing multi-task learning (MTL) performance in large language models, we proposed a novel framework that leverages a neural network to predict the best dataset combinations. The framework iteratively refines the selection, greatly improving efficiency, while being model-, dataset-, and domain-independent. Through experiments on 12 biomedical datasets across four tasks - named entity recognition, relation extraction, event extraction, and text classification-we demonstrate that our approach effectively identifies better combinations, even for tasks that may seem unpromising from a human perspective. This verifies that our framework provides a promising solution for maximizing MTL potential.