# MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces

arXiv: https://arxiv.org/abs/2109.04240 · HTML: https://mlsanity.com/paper/2109.04240/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise](https://mlsanity.com/paper/1802.05300/) (2018)
- [An introduction to domain adaptation and transfer learning](https://mlsanity.com/paper/1812.11806/) (2018)
- [Few-Shot Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.02536/) (2017)
- [Meta Label Correction for Noisy Label Learning](https://mlsanity.com/paper/1911.03809/) (2019)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Few-shot Domain Adaptation by Causal Mechanism Transfer](https://mlsanity.com/paper/2002.03497/) (2020)
- [Leveraging Multi-Source Weak Social Supervision for Early Detection of Fake News](https://mlsanity.com/paper/2004.01732/) (2020)
- [MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning](https://mlsanity.com/paper/2104.07908/) (2021)
- [Meta Multi-Task Learning for Sequence Modeling](https://mlsanity.com/paper/1802.08969/) (2018)
- [Learning with Weak Supervision for Email Intent Detection](https://mlsanity.com/paper/2005.13084/) (2020)
- [Multi-Task Learning for Sequence Tagging: An Empirical Study](https://mlsanity.com/paper/1808.04151/) (2018)
- [Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning](https://mlsanity.com/paper/1708.00993/) (2017)
- [Multi-task Learning of Pairwise Sequence Classification Tasks Over Disparate Label Spaces](https://mlsanity.com/paper/1802.09913/) (2018)
- [Multitask Learning for Fine-Grained Twitter Sentiment Analysis](https://mlsanity.com/paper/1707.03569/) (2017)

## Abstract (arXiv)

Albeit the universal representational power of pre-trained language models, adapting them onto a specific NLP task still requires a considerably large amount of labeled data. Effective task fine-tuning meets challenges when only a few labeled examples are present for the task. In this paper, we aim to the address of the problem of few shot task learning by exploiting and transferring from a different task which admits a related but disparate label space. Specifically, we devise a label transfer network (LTN) to transform the labels from source task to the target task of interest for training. Both the LTN and the model for task prediction are learned via a bi-level optimization framework, which we term as MetaXT. MetaXT offers a principled solution to best adapt a pre-trained language model to the target task by transferring knowledge from the source task. Empirical evaluations on cross-task transfer settings for four NLP tasks, from two different types of label space disparities, demonstrate the effectiveness of MetaXT, especially when the labeled data in the target task is limited.