# Unsupervised Text Generation by Learning from Search

arXiv: https://arxiv.org/abs/2007.08557 · HTML: https://mlsanity.com/paper/2007.08557/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 30 of 36)

- [A Survey of Text Watermarking in the Era of Large Language Models](https://mlsanity.com/paper/2312.07913/) (2023)
- [Learning to Model Editing Processes](https://mlsanity.com/paper/2205.12374/) (2022)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://mlsanity.com/paper/2110.10668/) (2021)
- [HICF: Hyperbolic Informative Collaborative Filtering](https://mlsanity.com/paper/2207.09051/) (2022)
- [ProQA: Structural Prompt-based Pre-training for Unified Question Answering](https://mlsanity.com/paper/2205.04040/) (2022)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [Text Revision by On-the-Fly Representation Optimization](https://mlsanity.com/paper/2204.07359/) (2022)
- [A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model](https://mlsanity.com/paper/2201.10707/) (2022)
- [Prompt-Based Editing for Text Style Transfer](https://mlsanity.com/paper/2301.11997/) (2023)
- [Generating Fluent Fact Checking Explanations with Unsupervised Post-Editing](https://mlsanity.com/paper/2112.06924/) (2021)
- [AEON: A Method for Automatic Evaluation of NLP Test Cases](https://mlsanity.com/paper/2205.06439/) (2022)
- [CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models](https://mlsanity.com/paper/2403.03514/) (2024)
- [A Call for Standardization and Validation of Text Style Transfer Evaluation](https://mlsanity.com/paper/2306.00539/) (2023)
- [Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization](https://mlsanity.com/paper/2205.14521/) (2022)
- [Non-Parallel Text Style Transfer with Self-Parallel Supervision](https://mlsanity.com/paper/2204.08123/) (2022)
- [Teacher Forcing Recovers Reward Functions for Text Generation](https://mlsanity.com/paper/2210.08708/) (2022)
- [Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations](https://mlsanity.com/paper/2305.14618/) (2023)
- [Graphine: A Dataset for Graph-aware Terminology Definition Generation](https://mlsanity.com/paper/2109.04018/) (2021)
- [Generic resources are what you need: Style transfer tasks without task-specific parallel training data](https://mlsanity.com/paper/2109.04543/) (2021)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Deep Structural Knowledge Exploitation and Synergy for Estimating Node Importance Value on Heterogeneous Information Networks](https://mlsanity.com/paper/2402.12411/) (2024)
- [Search and Learn: Improving Semantic Coverage for Data-to-Text Generation](https://mlsanity.com/paper/2112.02770/) (2021)
- [Graph Component Contrastive Learning for Concept Relatedness Estimation](https://mlsanity.com/paper/2206.12556/) (2022)
- [Improving Task Generalization via Unified Schema Prompt](https://mlsanity.com/paper/2208.03229/) (2022)
- [Simulated annealing for optimization of graphs and sequences](https://mlsanity.com/paper/2110.01384/) (2021)
- [Simulated Annealing for Emotional Dialogue Systems](https://mlsanity.com/paper/2109.10715/) (2021)
- [Stylistic Retrieval-based Dialogue System with Unparallel Training Data](https://mlsanity.com/paper/2109.05477/) (2021)
- [VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder](https://mlsanity.com/paper/2307.00852/) (2023)
- [The Emergence of Chunking Structures with Hierarchical RNN](https://mlsanity.com/paper/2309.04919/) (2023)
- [Search and Learning for Unsupervised Text Generation](https://mlsanity.com/paper/2309.09497/) (2023)

## Abstract (arXiv)

In this work, we present TGLS, a novel framework to unsupervised Text Generation by Learning from Search. We start by applying a strong search algorithm (in particular, simulated annealing) towards a heuristically defined objective that (roughly) estimates the quality of sentences. Then, a conditional generative model learns from the search results, and meanwhile smooth out the noise of search. The alternation between search and learning can be repeated for performance bootstrapping. We demonstrate the effectiveness of TGLS on two real-world natural language generation tasks, paraphrase generation and text formalization. Our model significantly outperforms unsupervised baseline methods in both tasks. Especially, it achieves comparable performance with the state-of-the-art supervised methods in paraphrase generation.