# MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective

arXiv: https://arxiv.org/abs/2210.14650 · HTML: https://mlsanity.com/paper/2210.14650/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [Strategies for Structuring Story Generation](https://mlsanity.com/paper/1902.01109/) (2019)
- [Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training](https://mlsanity.com/paper/1911.03860/) (2019)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [Planning with Learned Entity Prompts for Abstractive Summarization](https://mlsanity.com/paper/2104.07606/) (2021)
- [Contrastive Learning with Adversarial Perturbations for Conditional Text Generation](https://mlsanity.com/paper/2012.07280/) (2020)
- [Neural data-to-text generation: A comparison between pipeline and end-to-end architectures](https://mlsanity.com/paper/1908.09022/) (2019)
- [Argument Generation with Retrieval, Planning, and Realization](https://mlsanity.com/paper/1906.03717/) (2019)
- [Pre-training Text-to-Text Transformers for Concept-centric Common Sense](https://mlsanity.com/paper/2011.07956/) (2020)
- [PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation](https://mlsanity.com/paper/2010.02301/) (2020)
- [Knowledge-Grounded Dialogue Generation with a Unified Knowledge Representation](https://mlsanity.com/paper/2112.07924/) (2021)
- [PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation](https://mlsanity.com/paper/2203.09100/) (2022)
- [Negative Training for Neural Dialogue Response Generation](https://mlsanity.com/paper/1903.02134/) (2019)
- [CoNT: Contrastive Neural Text Generation](https://mlsanity.com/paper/2205.14690/) (2022)
- [Language modeling via stochastic processes](https://mlsanity.com/paper/2203.11370/) (2022)
- [DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer](https://mlsanity.com/paper/2110.05999/) (2021)
- [Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task](https://mlsanity.com/paper/2010.05141/) (2020)
- [DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation](https://mlsanity.com/paper/2106.00791/) (2021)
- [Paper Abstract Writing through Editing Mechanism](https://mlsanity.com/paper/1805.06064/) (2018)

## Built on by (most cited first, 4 of 4)

- [Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generation](https://mlsanity.com/paper/2406.19643/) (2024)
- [AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interaction](https://mlsanity.com/paper/2310.20352/) (2023)
- [Exploring Italian sentence embeddings properties through multi-tasking](https://mlsanity.com/paper/2409.06622/) (2024)
- [SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Models](https://mlsanity.com/paper/2402.00474/) (2024)

## Abstract (arXiv)

Teaching neural models to generate narrative coherent texts is a critical problem. Recent pre-trained language models have achieved promising results, but there is still a gap between human written texts and machine-generated outputs. In this work, we propose a novel multi-task training strategy for coherent text generation grounded on the cognitive theory of writing, which empowers the model to learn essential subskills needed for writing including planning and reviewing besides end-to-end generation. We extensively evaluate our model on three open-ended generation tasks including story generation, news article writing and argument generation. Experiments show that our model achieves better results on both few-shot and fully-supervised settings than strong baselines, and human evaluations confirm that our model can generate more coherent outputs.