# The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding

arXiv: https://arxiv.org/abs/2002.07972 · HTML: https://mlsanity.com/paper/2002.07972/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2020.

## Built on by (most cited first, 28 of 28)

- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin](https://mlsanity.com/paper/2312.09979/) (2023)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization](https://mlsanity.com/paper/2105.12002/) (2021)
- [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://mlsanity.com/paper/2303.04222/) (2023)
- [DeepStruct: Pretraining of Language Models for Structure Prediction](https://mlsanity.com/paper/2205.10475/) (2022)
- [Efficient Long Sequence Modeling via State Space Augmented Transformer](https://mlsanity.com/paper/2212.08136/) (2022)
- [METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals](https://mlsanity.com/paper/2204.06644/) (2022)
- [Protein Language Models and Structure Prediction: Connection and Progression](https://mlsanity.com/paper/2211.16742/) (2022)
- [No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models](https://mlsanity.com/paper/2202.02664/) (2022)
- [Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding](https://mlsanity.com/paper/2110.08419/) (2021)
- [CLUES: Few-Shot Learning Evaluation in Natural Language Understanding](https://mlsanity.com/paper/2111.02570/) (2021)
- [Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach](https://mlsanity.com/paper/2104.04886/) (2021)
- [CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing](https://mlsanity.com/paper/2204.06625/) (2022)
- [The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains](https://mlsanity.com/paper/2205.11416/) (2022)
- [Targeted Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/2104.05847/) (2021)
- [A Flexible Multi-Task Model for BERT Serving](https://mlsanity.com/paper/2107.05377/) (2021)
- [Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?](https://mlsanity.com/paper/2304.11520/) (2023)
- [College Student Retention Risk Analysis From Educational Database using Multi-Task Multi-Modal Neural Fusion](https://mlsanity.com/paper/2109.05178/) (2021)
- [ARCH: Efficient Adversarial Regularized Training with Caching](https://mlsanity.com/paper/2109.07048/) (2021)
- [Fast-ELECTRA for Efficient Pre-training](https://mlsanity.com/paper/2310.07347/) (2023)
- [OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction](https://mlsanity.com/paper/2105.05535/) (2021)
- [Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning](https://mlsanity.com/paper/2106.05221/) (2021)
- [OCHADAI at SemEval-2022 Task 2: Adversarial Training for Multilingual Idiomaticity Detection](https://mlsanity.com/paper/2206.03025/) (2022)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [Switchable Decision: Dynamic Neural Generation Networks](https://mlsanity.com/paper/2405.04513/) (2024)
- [Quantization Range Estimation for Convolutional Neural Networks](https://mlsanity.com/paper/2510.04044/) (2025)

## Abstract (arXiv)

We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pre-trained models will be publicly available at https://github.com/namisan/mt-dnn.