# A Million Tweets Are Worth a Few Points: Tuning Transformers for Customer Service Tasks

arXiv: https://arxiv.org/abs/2104.07944 · HTML: https://mlsanity.com/paper/2104.07944/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Automatically Identifying Complaints in Social Media](https://mlsanity.com/paper/1906.03890/) (2019)
- [COTA: Improving the Speed and Accuracy of Customer Support through Ranking and Deep Networks](https://mlsanity.com/paper/1807.01337/) (2018)
- [Machine Reading Comprehension for Answer Re-Ranking in Customer Support Chatbots](https://mlsanity.com/paper/1902.04574/) (2019)
- [Semi-Supervised Learning Approach to Discover Enterprise User Insights from Feedback and Support](https://mlsanity.com/paper/2007.09303/) (2020)

## Built on by (most cited first, 4 of 4)

- [EduQG: A Multi-format Multiple Choice Dataset for the Educational Domain](https://mlsanity.com/paper/2210.06104/) (2022)
- [TaDaa: real time Ticket Assignment Deep learning Auto Advisor for customer support, help desk, and issue ticketing systems](https://mlsanity.com/paper/2207.11187/) (2022)
- [EmoTwiCS: A Corpus for Modelling Emotion Trajectories in Dutch Customer Service Dialogues on Twitter](https://mlsanity.com/paper/2310.06536/) (2023)
- [A Customer Journey in the Land of Oz: Leveraging the Wizard of Oz Technique to Model Emotions in Customer Service Interactions](https://mlsanity.com/paper/2511.21909/) (2025)

## Abstract (arXiv)

In online domain-specific customer service applications, many companies struggle to deploy advanced NLP models successfully, due to the limited availability of and noise in their datasets. While prior research demonstrated the potential of migrating large open-domain pretrained models for domain-specific tasks, the appropriate (pre)training strategies have not yet been rigorously evaluated in such social media customer service settings, especially under multilingual conditions. We address this gap by collecting a multilingual social media corpus containing customer service conversations (865k tweets), comparing various pipelines of pretraining and finetuning approaches, applying them on 5 different end tasks. We show that pretraining a generic multilingual transformer model on our in-domain dataset, before finetuning on specific end tasks, consistently boosts performance, especially in non-English settings.