# On-Device Emoji Classifier Trained with GPT-based Data Augmentation for a Mobile Keyboard

arXiv: https://arxiv.org/abs/2411.05031 · HTML: https://mlsanity.com/paper/2411.05031/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Federated Learning for Emoji Prediction in a Mobile Keyboard](https://mlsanity.com/paper/1906.04329/) (2019)
- [Neural Data Augmentation via Example Extrapolation](https://mlsanity.com/paper/2102.01335/) (2021)
- [Octopus v2: On-device language model for super agent](https://mlsanity.com/paper/2404.01744/) (2024)
- [emoji2vec: Learning Emoji Representations from their Description](https://mlsanity.com/paper/1609.08359/) (2016)
- [Natural Language Processing with Small Feed-Forward Networks](https://mlsanity.com/paper/1708.00214/) (2017)
- [Are Emojis Predictable?](https://mlsanity.com/paper/1702.07285/) (2017)
- [EmojiLM: Modeling the New Emoji Language](https://mlsanity.com/paper/2311.01751/) (2023)
- [Emoji Prediction: Extensions and Benchmarking](https://mlsanity.com/paper/2007.07389/) (2020)
- [A Federated Approach to Predicting Emojis in Hindi Tweets](https://mlsanity.com/paper/2211.06401/) (2022)
- [Contrastive Learning of Emoji-based Representations for Resource-Poor Languages](https://mlsanity.com/paper/1804.01855/) (2018)
- [Federated Learning Based Multilingual Emoji Prediction In Clean and Attack Scenarios](https://mlsanity.com/paper/2304.01005/) (2023)
- [Predict Emoji Combination with Retrieval Strategy](https://mlsanity.com/paper/1908.07761/) (2019)
- [VoiceMoji: A Novel On-Device Pipeline for Seamless Emoji Insertion in Dictation](https://mlsanity.com/paper/2112.12028/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Distributed Generative AI Approach for Heterogeneous Multi-Domain Environments under Data Sharing constraints](https://mlsanity.com/paper/2507.12979/) (2025)
- [FLOP-Efficient Training: Early Stopping Based on Test-Time Compute Awareness](https://mlsanity.com/paper/2601.01332/) (2026)

## Abstract (arXiv)

Emojis improve communication quality among smart-phone users that use mobile keyboards to exchange text. To predict emojis for users based on input text, we should consider the on-device low memory and time constraints, ensure that the on-device emoji classifier covers a wide range of emoji classes even though the emoji dataset is typically imbalanced, and adapt the emoji classifier output to user favorites. This paper proposes an on-device emoji classifier based on MobileBert with reasonable memory and latency requirements for SwiftKey. To account for the data imbalance, we utilize the widely used GPT to generate one or more tags for each emoji class. For each emoji and corresponding tags, we merge the original set with GPT-generated sentences and label them with this emoji without human intervention to alleviate the data imbalance. At inference time, we interpolate the emoji output with the user history for emojis for better emoji classifications. Results show that the proposed on-device emoji classifier deployed for SwiftKey increases the accuracy performance of emoji prediction particularly on rare emojis and emoji engagement.