# A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks

arXiv: https://arxiv.org/abs/1811.06031 · HTML: https://mlsanity.com/paper/1811.06031/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2018.

## Built on by (most cited first, 30 of 54)

- [A Comprehensive Survey on Graph Anomaly Detection with Deep Learning](https://mlsanity.com/paper/2106.07178/) (2021)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks](https://mlsanity.com/paper/2303.02483/) (2023)
- [An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning](https://mlsanity.com/paper/2102.05980/) (2021)
- [Towards Versatile Embodied Navigation](https://mlsanity.com/paper/2210.16822/) (2022)
- [Specializing Multilingual Language Models: An Empirical Study](https://mlsanity.com/paper/2106.09063/) (2021)
- [Multi-task Active Learning for Pre-trained Transformer-based Models](https://mlsanity.com/paper/2208.05379/) (2022)
- [HoME: Hierarchy of Multi-Gate Experts for Multi-Task Learning at Kuaishou](https://mlsanity.com/paper/2408.05430/) (2024)
- [Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification](https://mlsanity.com/paper/2108.02598/) (2021)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers](https://mlsanity.com/paper/2102.06008/) (2021)
- [Balancing Average and Worst-case Accuracy in Multitask Learning](https://mlsanity.com/paper/2110.05838/) (2021)
- [Let the Model Decide its Curriculum for Multitask Learning](https://mlsanity.com/paper/2205.09898/) (2022)
- [Beyond Rule-based Named Entity Recognition and Relation Extraction for Process Model Generation from Natural Language Text](https://mlsanity.com/paper/2305.03960/) (2023)
- [Boosting Crop Classification by Hierarchically Fusing Satellite, Rotational, and Contextual Data](https://mlsanity.com/paper/2305.12011/) (2023)
- [AntLM: Bridging Causal and Masked Language Models](https://mlsanity.com/paper/2412.03275/) (2024)
- [Multilingual Relation Classification via Efficient and Effective Prompting](https://mlsanity.com/paper/2210.13838/) (2022)
- [Human Mobility Prediction with Causal and Spatial-constrained Multi-task Network](https://mlsanity.com/paper/2206.05731/) (2022)
- [RobustAnalog: Fast Variation-Aware Analog Circuit Design Via Multi-task RL](https://mlsanity.com/paper/2207.06412/) (2022)
- [Better Call the Plumber: Orchestrating Dynamic Information Extraction Pipelines](https://mlsanity.com/paper/2102.10966/) (2021)
- [Multimodal Crop Type Classification Fusing Multi-Spectral Satellite Time Series with Farmers Crop Rotations and Local Crop Distribution](https://mlsanity.com/paper/2208.10838/) (2022)
- [One Semantic Parser to Parse Them All: Sequence to Sequence Multi-Task Learning on Semantic Parsing Datasets](https://mlsanity.com/paper/2106.04476/) (2021)
- [Exceeding the Limits of Visual-Linguistic Multi-Task Learning](https://mlsanity.com/paper/2107.13054/) (2021)
- [Analyzing Multi-Task Learning for Abstractive Text Summarization](https://mlsanity.com/paper/2210.14606/) (2022)
- [DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2305.17305/) (2023)
- [CONTRASTE: Supervised Contrastive Pre-training With Aspect-based Prompts For Aspect Sentiment Triplet Extraction](https://mlsanity.com/paper/2310.15577/) (2023)

## Abstract (arXiv)

Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.