# SkillNet-NLG: General-Purpose Natural Language Generation with a Sparsely Activated Approach

arXiv: https://arxiv.org/abs/2204.12184 · HTML: https://mlsanity.com/paper/2204.12184/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation](https://mlsanity.com/paper/2004.04100/) (2020)
- [MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization](https://mlsanity.com/paper/2004.12302/) (2020)
- [SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understanding](https://mlsanity.com/paper/2203.03312/) (2022)
- [A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graph](https://mlsanity.com/paper/2010.05511/) (2020)

## Built on by (most cited first, 3 of 3)

- [One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code](https://mlsanity.com/paper/2205.06126/) (2022)
- [Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers](https://mlsanity.com/paper/2210.11265/) (2022)
- [SkillNet-X: A Multilingual Multitask Model with Sparsely Activated Skills](https://mlsanity.com/paper/2306.16176/) (2023)

## Abstract (arXiv)

We present SkillNet-NLG, a sparsely activated approach that handles many natural language generation tasks with one model. Different from traditional dense models that always activate all the parameters, SkillNet-NLG selectively activates relevant parts of the parameters to accomplish a task, where the relevance is controlled by a set of predefined skills. The strength of such model design is that it provides an opportunity to precisely adapt relevant skills to learn new tasks effectively. We evaluate on Chinese natural language generation tasks. Results show that, with only one model file, SkillNet-NLG outperforms previous best performance methods on four of five tasks. SkillNet-NLG performs better than two multi-task learning baselines (a dense model and a Mixture-of-Expert model) and achieves comparable performance to task-specific models. Lastly, SkillNet-NLG surpasses baseline systems when being adapted to new tasks.