# When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP

arXiv: https://arxiv.org/abs/2305.14007 · HTML: https://mlsanity.com/paper/2305.14007/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 58 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models](https://mlsanity.com/paper/2010.05874/) (2020)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining](https://mlsanity.com/paper/2005.02799/) (2020)
- [Financial Document Causality Detection Shared Task (FinCausal 2020)](https://mlsanity.com/paper/2012.02505/) (2020)
- [Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning](https://mlsanity.com/paper/2204.11117/) (2022)
- [What's in your Head? Emergent Behaviour in Multi-Task Transformer Models](https://mlsanity.com/paper/2104.06129/) (2021)
- [Numeral Understanding in Financial Tweets for Fine-grained Crowd-based Forecasting](https://mlsanity.com/paper/1809.05356/) (2018)
- [Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?](https://mlsanity.com/paper/2212.06645/) (2022)

## Built on by (most cited first, 5 of 5)

- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [IL-PCSR: Legal Corpus for Prior Case and Statute Retrieval](https://mlsanity.com/paper/2511.00268/) (2025)
- [Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information](https://mlsanity.com/paper/2410.12774/) (2024)
- [Form Over Content In Gradient-Based Data Attribution Methods](https://mlsanity.com/paper/2609.19589/) (2026)
- [HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts](https://mlsanity.com/paper/2608.29790/) (2026)

## Abstract (arXiv)

Multi-task learning (MTL) aims at achieving a better model by leveraging data and knowledge from multiple tasks. However, MTL does not always work -- sometimes negative transfer occurs between tasks, especially when aggregating loosely related skills, leaving it an open question when MTL works. Previous studies show that MTL performance can be improved by algorithmic tricks. However, what tasks and skills should be included is less well explored. In this work, we conduct a case study in Financial NLP where multiple datasets exist for skills relevant to the domain, such as numeric reasoning and sentiment analysis. Due to the task difficulty and data scarcity in the Financial NLP domain, we explore when aggregating such diverse skills from multiple datasets with MTL can work. Our findings suggest that the key to MTL success lies in skill diversity, relatedness between tasks, and choice of aggregation size and shared capacity. Specifically, MTL works well when tasks are diverse but related, and when the size of the task aggregation and the shared capacity of the model are balanced to avoid overwhelming certain tasks.