# Collocation2Text: Controllable Text Generation from Guide Phrases in Russian

arXiv: https://arxiv.org/abs/2206.09248 · HTML: https://mlsanity.com/paper/2206.09248/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 18 Jun 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [Artificial Intelligence in the Creative Industries: A Review](https://mlsanity.com/paper/2007.12391/) (2020)
- [Exploring Controllable Text Generation Techniques](https://mlsanity.com/paper/2005.01822/) (2020)
- [Story Generation from Sequence of Independent Short Descriptions](https://mlsanity.com/paper/1707.05501/) (2017)
- [A Survey of Natural Language Generation Techniques with a Focus on Dialogue Systems - Past, Present and Future Directions](https://mlsanity.com/paper/1906.00500/) (2019)
- [Morphological Analyzer and Generator for Russian and Ukrainian Languages](https://mlsanity.com/paper/1503.07283/) (2015)
- [Fact-Enhanced Synthetic News Generation](https://mlsanity.com/paper/2012.04778/) (2020)
- [Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation](https://mlsanity.com/paper/1901.00398/) (2019)
- [Narrative Planning: Compilations to Classical Planning](https://mlsanity.com/paper/1401.5856/) (2014)

## Abstract (arXiv)

Large pre-trained language models are capable of generating varied and fluent texts. Starting from the prompt, these models generate a narrative that can develop unpredictably. The existing methods of controllable text generation, which guide the narrative in the text in the user-specified direction, require creating a training corpus and an additional time-consuming training procedure. The paper proposes and investigates Collocation2Text, a plug-and-play method for automatic controllable text generation in Russian, which does not require fine-tuning. The method is based on two interacting models: the autoregressive language ruGPT-3 model and the autoencoding language ruRoBERTa model. The idea of the method is to shift the output distribution of the autoregressive model according to the output distribution of the autoencoding model in order to ensure a coherent transition of the narrative in the text towards the guide phrase, which can contain single words or collocations. The autoencoding model, which is able to take into account the left and right contexts of the token, "tells" the autoregressive model which tokens are the most and least logical at the current generation step, increasing or decreasing the probabilities of the corresponding tokens. The experiments on generating news articles using the proposed method showed its effectiveness for automatically generated fluent texts which contain coherent transitions between user-specified phrases.