# Improving Customer Service Chatbots with Attention-based Transfer Learning

arXiv: https://arxiv.org/abs/2111.14621 · HTML: https://mlsanity.com/paper/2111.14621/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 41 references that are in the map.
- First posted on arXiv 24 Nov 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce](https://mlsanity.com/paper/1711.08726/) (2017)
- [Pretrained Transformers for Simple Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2001.11985/) (2020)
- [Goal-Oriented Chatbot Dialog Management Bootstrapping with Transfer Learning](https://mlsanity.com/paper/1802.00500/) (2018)
- [Ensemble-Based Deep Reinforcement Learning for Chatbots](https://mlsanity.com/paper/1908.10422/) (2019)
- [Chatbot Interaction with Artificial Intelligence: Human Data Augmentation with T5 and Language Transformer Ensemble for Text Classification](https://mlsanity.com/paper/2010.05990/) (2020)

## Built on by (most cited first, 1 of 1)

- [Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation](https://mlsanity.com/paper/2602.00665/) (2026)

## Abstract (arXiv)

With growing societal acceptance and increasing cost efficiency due to mass production, service robots are beginning to cross from the industrial to the social domain. Currently, customer service robots tend to be digital and emulate social interactions through on-screen text, but state-of-the-art research points towards physical robots soon providing customer service in person. This article explores two possibilities. Firstly, whether transfer learning can aid in the improvement of customer service chatbots between business domains. Second, the implementation of a framework for physical robots for in-person interaction. Modelled on social interaction with Twitter customer support accounts, transformer-based chatbot models are initially assigned to learn one domain from an initial random weight distribution. Given shared vocabulary, each model is then tasked with learning another domain by transferring knowledge from the previous. Following studies on 19 different businesses, results show that the majority of models are improved when transferring weights from at least one other domain, in particular those that are more data-scarce than others. General language transfer learning occurs, as well as higher-level transfer of similar domain knowledge, in several cases. The chatbots are finally implemented on Temi and Pepper robots, with feasibility issues encountered and solutions are proposed to overcome them.