# Beyond Shared Hierarchies: Deep Multitask Learning through Soft Layer Ordering

arXiv: https://arxiv.org/abs/1711.00108 · HTML: https://mlsanity.com/paper/1711.00108/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2017.

## Built on by (most cited first, 28 of 28)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Continual Learning via Local Module Composition](https://mlsanity.com/paper/2111.07736/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning](https://mlsanity.com/paper/2408.07057/) (2024)
- [Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning](https://mlsanity.com/paper/2302.11289/) (2023)
- [Modular Lifelong Reinforcement Learning via Neural Composition](https://mlsanity.com/paper/2207.00429/) (2022)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Neural Relational Inference with Fast Modular Meta-learning](https://mlsanity.com/paper/2310.07015/) (2023)
- [Dynamic Neural Network for Multi-Task Learning Searching across Diverse Network Topologies](https://mlsanity.com/paper/2303.06856/) (2023)
- [Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanities](https://mlsanity.com/paper/2310.19626/) (2023)
- [Compositional Learning-based Planning for Vision POMDPs](https://mlsanity.com/paper/2112.09456/) (2021)
- [Entangled Kernels -- Beyond Separability](https://mlsanity.com/paper/2101.05514/) (2021)
- [MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters](https://mlsanity.com/paper/2311.04251/) (2023)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Multi-Order Networks for Action Unit Detection](https://mlsanity.com/paper/2202.00446/) (2022)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Towards Anytime Retrieval: A Benchmark for Anytime Person Re-Identification](https://mlsanity.com/paper/2509.16635/) (2025)
- [Glider: Global and Local Instruction-Driven Expert Router](https://mlsanity.com/paper/2410.07172/) (2024)
- [Feature Pyramid Network for Multi-task Affective Analysis](https://mlsanity.com/paper/2107.03670/) (2021)
- [Development of a deep learning-based tool to assist wound classification](https://mlsanity.com/paper/2303.16522/) (2023)
- [Visual Attention Prompted Prediction and Learning](https://mlsanity.com/paper/2310.08420/) (2023)
- [Fast Line Search for Multi-Task Learning](https://mlsanity.com/paper/2110.00874/) (2021)
- [Gesture Recognition with a Skeleton-Based Keyframe Selection Module](https://mlsanity.com/paper/2112.01736/) (2021)
- [Distributed Continual Learning](https://mlsanity.com/paper/2405.17466/) (2024)
- [Toward Robust EEG-based Intention Decoding during Misarticulated Speech in Dysarthria](https://mlsanity.com/paper/2511.07895/) (2025)
- [MTMed3D: A Multi-Task Transformer-Based Model for 3D Medical Imaging](https://mlsanity.com/paper/2511.12373/) (2025)

## Abstract (arXiv)

Existing deep multitask learning (MTL) approaches align layers shared between tasks in a parallel ordering. Such an organization significantly constricts the types of shared structure that can be learned. The necessity of parallel ordering for deep MTL is first tested by comparing it with permuted ordering of shared layers. The results indicate that a flexible ordering can enable more effective sharing, thus motivating the development of a soft ordering approach, which learns how shared layers are applied in different ways for different tasks. Deep MTL with soft ordering outperforms parallel ordering methods across a series of domains. These results suggest that the power of deep MTL comes from learning highly general building blocks that can be assembled to meet the demands of each task.