# Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning

arXiv: https://arxiv.org/abs/2110.06273 · HTML: https://mlsanity.com/paper/2110.06273/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Is Multilingual BERT Fluent in Language Generation?](https://mlsanity.com/paper/1910.03806/) (2019)
- [Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation](https://mlsanity.com/paper/1809.00794/) (2018)
- [DLGNet: A Transformer-based Model for Dialogue Response Generation](https://mlsanity.com/paper/1908.01841/) (2019)

## Built on by (most cited first, 11 of 11)

- [Leveraging Implicit Feedback from Deployment Data in Dialogue](https://mlsanity.com/paper/2307.14117/) (2023)
- [MDIA: A Benchmark for Multilingual Dialogue Generation in 46 Languages](https://mlsanity.com/paper/2208.13078/) (2022)
- [HaT5: Hate Language Identification using Text-to-Text Transfer Transformer](https://mlsanity.com/paper/2202.05690/) (2022)
- [State-of-the-art in Open-domain Conversational AI: A Survey](https://mlsanity.com/paper/2205.00965/) (2022)
- [T5 for Hate Speech, Augmented Data and Ensemble](https://mlsanity.com/paper/2210.05480/) (2022)
- [Vector Representations of Idioms in Conversational Systems](https://mlsanity.com/paper/2205.03666/) (2022)
- [Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generation](https://mlsanity.com/paper/2305.07393/) (2023)
- [ML_LTU at SemEval-2022 Task 4: T5 Towards Identifying Patronizing and Condescending Language](https://mlsanity.com/paper/2204.07432/) (2022)
- [NLP-LTU at SemEval-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Dataset](https://mlsanity.com/paper/2304.12847/) (2023)
- [AfriWOZ: Corpus for Exploiting Cross-Lingual Transferability for Generation of Dialogues in Low-Resource, African Languages](https://mlsanity.com/paper/2204.08083/) (2022)
- [Aligning a medium-size GPT model in English to a small closed domain in Spanish](https://mlsanity.com/paper/2303.17649/) (2023)

## Abstract (arXiv)

Building open-domain conversational systems (or chatbots) that produce convincing responses is a recognized challenge. Recent state-of-the-art (SoTA) transformer-based models for the generation of natural language dialogue have demonstrated impressive performance in simulating human-like, single-turn conversations in English. This work investigates, by an empirical study, the potential for transfer learning of such models to Swedish language. DialoGPT, an English language pre-trained model, is adapted by training on three different Swedish language conversational datasets obtained from publicly available sources. Perplexity score (an automated intrinsic language model metric) and surveys by human evaluation were used to assess the performances of the fine-tuned models, with results that indicate that the capacity for transfer learning can be exploited with considerable success. Human evaluators asked to score the simulated dialogue judged over 57% of the chatbot responses to be human-like for the model trained on the largest (Swedish) dataset. We provide the demos and model checkpoints of our English and Swedish chatbots on the HuggingFace platform for public use.