Data to 7 Oct 2026

A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

arXiv 1611.01587 ↗First posted 5 Nov 2016cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 13; 2023: 13; 2024: 4; 2025: 4; 2026: 1.

62 papers.

Sort:

By area:

2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
2022Multi-Task Learning as a Bargaining Gamecited by 209
2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
2021Multi-Task Learning in Natural Language Processing: An Overviewcited by 61
2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
2021A Survey on Green Deep Learningcited by 52
2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 50
2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43

38 more

2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
2021Annotating Columns with Pre-trained Language Modelscited by 37
2023Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidancecited by 35
2023Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithmscited by 33
2021Deep Multi-Task Learning for Joint Localization, Perception, and Predictioncited by 22
2021Specializing Multilingual Language Models: An Empirical Studycited by 21
2021Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biasescited by 21
2021Collaborative Intelligence: Challenges and Opportunitiescited by 18
2021Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Textcited by 14
2022Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approachcited by 14
2021PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsingcited by 13
2021Meta-Learning for Effective Multi-task and Multilingual Modellingcited by 13
2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 13
2024Supervised Fine-tuning in turn Improves Visual Foundation Modelscited by 13
2021Linguistically-Enriched and Context-Aware Zero-shot Slot Fillingcited by 12
2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
2023Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detectioncited by 10
2021What's in your Head? Emergent Behaviour in Multi-Task Transformer Modelscited by 9
2021PVG at WASSA 2021: A Multi-Input, Multi-Task, Transformer-Based Architecture for Empathy and Distress Predictioncited by 7
2021Learning to Generalize Compositionally by Transferring Across Semantic Parsing Taskscited by 6
2023Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Modelcited by 5
2024Cross-domain Named Entity Recognition via Graph Matchingcited by 5
2021Attention Guided Dialogue State Tracking with Sparse Supervisioncited by 4
2023A Novel Computational and Modeling Foundation for Automatic Coherence Assessmentcited by 4
2021Instance-Based Neural Dependency Parsingcited by 3
2023Decomposition Ascribed Synergistic Learning for Unified Image Restorationcited by 3
2021HierMUD: Hierarchical Multi-task Unsupervised Domain Adaptation between Bridges for Drive-by Damage Diagnosiscited by 2
2022Learning with Latent Structures in Natural Language Processing: A Surveycited by 2
2022Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Taskcited by 2
2022Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5cited by 2
2021UPB at SemEval-2021 Task 5: Virtual Adversarial Training for Toxic Spans Detectioncited by 1
2021Out-of-Scope Domain and Intent Classification through Hierarchical Joint Modelingcited by 1
2021Predictive Representation Learning for Language Modelingcited by 1
2021ClueReader: Heterogeneous Graph Attention Network for Multi-hop Machine Reading Comprehensioncited by 1
2021Traffic Event Detection as a Slot Filling Problemcited by 1
2022Medical Knowledge Graph QA for Drug-Drug Interaction Prediction based on Multi-hop Machine Reading Comprehensioncited by 1
2023Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Rankingcited by 1

50 most cited of 62 All 62 (JSON)