# Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling

arXiv: https://arxiv.org/abs/2102.03551 · HTML: https://mlsanity.com/paper/2102.03551/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 6 Feb 2021.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Unsupervised Neural Machine Translation](https://mlsanity.com/paper/1710.11041/) (2017)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [Phrase-Based & Neural Unsupervised Machine Translation](https://mlsanity.com/paper/1804.07755/) (2018)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/1902.10461/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://mlsanity.com/paper/1904.03396/) (2019)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [GPT-too: A language-model-first approach for AMR-to-text generation](https://mlsanity.com/paper/2005.09123/) (2020)
- [Language Model Prior for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2004.14928/) (2020)
- [Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue](https://mlsanity.com/paper/1906.07220/) (2019)
- [Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity](https://mlsanity.com/paper/2004.06577/) (2020)
- [Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings](https://mlsanity.com/paper/1606.05491/) (2016)
- [Revisiting Challenges in Data-to-Text Generation with Fact Grounding](https://mlsanity.com/paper/2001.03830/) (2020)
- [It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information](https://mlsanity.com/paper/2005.02354/) (2020)
- [Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence](https://mlsanity.com/paper/2005.01096/) (2020)
- [A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation](https://mlsanity.com/paper/1805.06553/) (2018)
- [Fidelity-Weighted Learning](https://mlsanity.com/paper/1711.02799/) (2017)
- [Dual Supervised Learning for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.06196/) (2019)

## Built on by (most cited first, 11 of 11)

- [On Training Instance Selection for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2107.03176/) (2021)
- [ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues](https://mlsanity.com/paper/2109.00430/) (2021)
- [Time-Aware Ancient Chinese Text Translation and Inference](https://mlsanity.com/paper/2107.03179/) (2021)
- [The SelectGen Challenge: Finding the Best Training Samples for Few-Shot Neural Text Generation](https://mlsanity.com/paper/2108.06614/) (2021)
- [Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation](https://mlsanity.com/paper/2108.12582/) (2021)
- [Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation](https://mlsanity.com/paper/2112.06240/) (2021)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model](https://mlsanity.com/paper/2207.06839/) (2022)
- [Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation](https://mlsanity.com/paper/2301.02275/) (2023)
- [Contextual Knowledge Learning For Dialogue Generation](https://mlsanity.com/paper/2305.18200/) (2023)
- [CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization](https://mlsanity.com/paper/2503.05750/) (2025)

## Abstract (arXiv)

Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive. Recent frameworks address this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expert-curated and the rest of the data is automatically annotated. We follow that approach, by automatically constructing a large-scale weakly-labeled data with a fine-tuned GPT-2, and employ a semi-supervised framework to jointly train the NLG and NLU models. The proposed framework adapts the parameter updates to the models according to the estimated label-quality. On both the E2E and Weather benchmarks, we show that this weakly supervised training paradigm is an effective approach under low resource scenarios and outperforming benchmark systems on both datasets when 100% of training data is used.