# Partially Randomizing Transformer Weights for Dialogue Response Diversity

arXiv: https://arxiv.org/abs/2311.10943 · HTML: https://mlsanity.com/paper/2311.10943/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 18 Nov 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Robust and Generalizable Visual Representation Learning via Random Convolutions](https://mlsanity.com/paper/2007.13003/) (2020)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [Learning to Write with Cooperative Discriminators](https://mlsanity.com/paper/1805.06087/) (2018)
- [Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training](https://mlsanity.com/paper/1911.03860/) (2019)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [Exploiting Persona Information for Diverse Generation of Conversational Responses](https://mlsanity.com/paper/1905.12188/) (2019)
- [Variational Transformers for Diverse Response Generation](https://mlsanity.com/paper/2003.12738/) (2020)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Guiding Variational Response Generator to Exploit Persona](https://mlsanity.com/paper/1911.02390/) (2019)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Random Vector Functional Link Neural Network based Ensemble Deep Learning](https://mlsanity.com/paper/1907.00350/) (2019)
- [Random Vector Functional Link Networks for Function Approximation on Manifolds](https://mlsanity.com/paper/2007.15776/) (2020)
- [Reservoir Transformers](https://mlsanity.com/paper/2012.15045/) (2020)
- [Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders](https://mlsanity.com/paper/2106.03410/) (2021)
- [A Hierarchical Latent Structure for Variational Conversation Modeling](https://mlsanity.com/paper/1804.03424/) (2018)
- [A Discrete CVAE for Response Generation on Short-Text Conversation](https://mlsanity.com/paper/1911.09845/) (2019)
- [Improving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agents](https://mlsanity.com/paper/2202.05971/) (2022)
- [Improving Variational Encoder-Decoders in Dialogue Generation](https://mlsanity.com/paper/1802.02032/) (2018)
- [Generating Multiple Diverse Responses for Short-Text Conversation](https://mlsanity.com/paper/1811.05696/) (2018)
- [Emotion-Regularized Conditional Variational Autoencoder for Emotional Response Generation](https://mlsanity.com/paper/2104.08857/) (2021)

## Built on by (most cited first, 1 of 1)

- [A survey of diversity quantification in natural language processing: The why, what, where and how](https://mlsanity.com/paper/2507.20858/) (2025)

## Abstract (arXiv)

Despite recent progress in generative open-domain dialogue, the issue of low response diversity persists. Prior works have addressed this issue via either novel objective functions, alternative learning approaches such as variational frameworks, or architectural extensions such as the Randomized Link (RL) Transformer. However, these approaches typically entail either additional difficulties during training/inference, or a significant increase in model size and complexity. Hence, we propose the underlinePartially underlineRandomized transunderlineFormer (PaRaFormer), a simple extension of the transformer which involves freezing the weights of selected layers after random initialization. Experimental results reveal that the performance of the PaRaformer is comparable to that of the aforementioned approaches, despite not entailing any additional training difficulty or increase in model complexity.