# Textual Data Distributions: Kullback Leibler Textual Distributions Contrasts on GPT-2 Generated Texts, with Supervised, Unsupervised Learning on Vaccine & Market Topics & Sentiment

arXiv: https://arxiv.org/abs/2107.02025 · HTML: https://mlsanity.com/paper/2107.02025/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 54 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [Logical Natural Language Generation from Open-Domain Tables](https://mlsanity.com/paper/2004.10404/) (2020)
- [Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding](https://mlsanity.com/paper/1807.01554/) (2018)
- [Adversarial Feature Matching for Text Generation](https://mlsanity.com/paper/1706.03850/) (2017)
- [Natural Language Processing Advancements By Deep Learning: A Survey](https://mlsanity.com/paper/2003.01200/) (2020)
- [Comparing BERT against traditional machine learning text classification](https://mlsanity.com/paper/2005.13012/) (2020)
- [Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs](https://mlsanity.com/paper/1812.04718/) (2018)
- [An Introductory Survey on Attention Mechanisms in NLP Problems](https://mlsanity.com/paper/1811.05544/) (2018)
- [Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNet](https://mlsanity.com/paper/2102.08036/) (2021)
- [COVID-19 Public Sentiment Insights and Machine Learning for Tweets Classification](https://mlsanity.com/paper/2005.10898/) (2020)
- [Unsupervised Contextual Paraphrase Generation using Lexical Control and Reinforcement Learning](https://mlsanity.com/paper/2103.12777/) (2021)
- [An Overview of Natural Language State Representation for Reinforcement Learning](https://mlsanity.com/paper/2007.09774/) (2020)
- [Strategies for Democratization of Supercomputing: Availability, Accessibility and Usability of High Performance Computing for Education and Practice of Big Data Analytics](https://mlsanity.com/paper/2104.09091/) (2021)

## Built on by (most cited first, 2 of 2)

- [De-identification is not enough: a comparison between de-identified and synthetic clinical notes](https://mlsanity.com/paper/2402.00179/) (2024)
- [Adaptive cognitive fit: Artificial intelligence augmented management of information facets and representations](https://mlsanity.com/paper/2204.11405/) (2022)

## Abstract (arXiv)

Efficient textual data distributions (TDD) alignment and generation are open research problems in textual analytics and NLP. It is presently difficult to parsimoniously and methodologically confirm that two or more natural language datasets belong to similar distributions, and to identify the extent to which textual data possess alignment. This study focuses on addressing a segment of the broader problem described above by applying multiple supervised and unsupervised machine learning (ML) methods to explore the behavior of TDD by (i) topical alignment, and (ii) by sentiment alignment. Furthermore we use multiple text generation methods including fine-tuned GPT-2, to generate text by topic and by sentiment. Finally we develop a unique process driven variation of Kullback-Leibler divergence (KLD) application to TDD, named KL Textual Distributions Contrasts(KL-TDC) to identify the alignment of machine generated textual corpora with naturally occurring textual corpora. This study thus identifies a unique approach for generating and validating TDD by topic and sentiment, which can be used to help address sparse data problems and other research, practice and classroom situations in need of artificially generated topic or sentiment aligned textual data.