# Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation

arXiv: https://arxiv.org/abs/2102.07635 · HTML: https://mlsanity.com/paper/2102.07635/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 15 of 15)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search](https://mlsanity.com/paper/2001.04246/) (2020)
- [VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification](https://mlsanity.com/paper/2004.05707/) (2020)
- [Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning](https://mlsanity.com/paper/2004.14074/) (2020)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [Revisiting Knowledge Distillation via Label Smoothing Regularization](https://mlsanity.com/paper/1909.11723/) (2019)

## Abstract (arXiv)

In pickup and delivery services, transaction classification based on customer provided free text is a challenging problem. It involves the association of a wide variety of customer inputs to a fixed set of categories while adapting to the various customer writing styles. This categorization is important for the business: it helps understand the market needs and trends, and also assist in building a personalized experience for different segments of the customers. Hence, it is vital to capture these category information trends at scale, with high precision and recall. In this paper, we focus on a specific use-case where a single category drives each transaction. We propose a cost-effective transaction classification approach based on semi-supervision and knowledge distillation frameworks. The approach identifies the category of a transaction using free text input given by the customer. We use weak labelling and notice that the performance gains are similar to that of using human-annotated samples. On a large internal dataset and on 20Newsgroup dataset, we see that RoBERTa performs the best for the categorization tasks. Further, using an ALBERT model (it has 33x fewer parameters vis-a-vis parameters of RoBERTa), with RoBERTa as the Teacher, we see a performance similar to that of RoBERTa and better performance over unadapted ALBERT. This framework, with ALBERT as a student and RoBERTa as teacher, is further referred to as R-ALBERT in this paper. The model is in production and is used by business to understand changing trends and take appropriate decisions.