# Meta learning with language models: Challenges and opportunities in the classification of imbalanced text

arXiv: https://arxiv.org/abs/2310.15019 · HTML: https://mlsanity.com/paper/2310.15019/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [Understanding Abuse: A Typology of Abusive Language Detection Subtasks](https://mlsanity.com/paper/1705.09899/) (2017)
- [Towards generalisable hate speech detection: a review on obstacles and solutions](https://mlsanity.com/paper/2102.08886/) (2021)
- [TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis](https://mlsanity.com/paper/2010.11091/) (2020)
- [Solo or Ensemble? Choosing a CNN Architecture for Melanoma Classification](https://mlsanity.com/paper/1904.12724/) (2019)
- [Can you tell? SSNet -- a Sagittal Stratum-inspired Neural Network Framework for Sentiment Analysis](https://mlsanity.com/paper/2006.12958/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Comprehensive Survey on Imbalanced Data Learning](https://mlsanity.com/paper/2502.08960/) (2025)

## Abstract (arXiv)

Detecting out of policy speech (OOPS) content is important but difficult. While machine learning is a powerful tool to tackle this challenging task, it is hard to break the performance ceiling due to factors like quantity and quality limitations on training data and inconsistencies in OOPS definition and data labeling. To realize the full potential of available limited resources, we propose a meta learning technique (MLT) that combines individual models built with different text representations. We analytically show that the resulting technique is numerically stable and produces reasonable combining weights. We combine the MLT with a threshold-moving (TM) technique to further improve the performance of the combined predictor on highly-imbalanced in-distribution and out-of-distribution datasets. We also provide computational results to show the statistically significant advantages of the proposed MLT approach. All authors contributed equally to this work.