# A multi-model-based deep learning framework for short text multiclass classification with the imbalanced and extremely small data set

arXiv: https://arxiv.org/abs/2206.12027 · HTML: https://mlsanity.com/paper/2206.12027/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 32 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Text Classification Algorithms: A Survey](https://mlsanity.com/paper/1904.08067/) (2019)
- [Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference](https://mlsanity.com/paper/2002.04815/) (2020)
- [Prompt-Learning for Short Text Classification](https://mlsanity.com/paper/2202.11345/) (2022)
- [Efficient, Uncertainty-based Moderation of Neural Networks Text Classifiers](https://mlsanity.com/paper/2204.01334/) (2022)
- [What Are People Asking About COVID-19? A Question Classification Dataset](https://mlsanity.com/paper/2005.12522/) (2020)
- [INT8 Winograd Acceleration for Conv1D Equipped ASR Models Deployed on Mobile Devices](https://mlsanity.com/paper/2010.14841/) (2020)

## Built on by (most cited first, 1 of 1)

- [Explaining Search Result Stances to Opinionated People](https://mlsanity.com/paper/2309.08460/) (2023)

## Abstract (arXiv)

Text classification plays an important role in many practical applications. In the real world, there are extremely small datasets. Most existing methods adopt pre-trained neural network models to handle this kind of dataset. However, these methods are either difficult to deploy on mobile devices because of their large output size or cannot fully extract the deep semantic information between phrases and clauses. This paper proposes a multimodel-based deep learning framework for short-text multiclass classification with an imbalanced and extremely small data set. Our framework mainly includes five layers: The encoder layer uses DISTILBERT to obtain context-sensitive dynamic word vectors that are difficult to represent in traditional feature engineering methods. Since the transformer part of this layer is distilled, our framework is compressed. Then, we use the next two layers to extract deep semantic information. The output of the encoder layer is sent to a bidirectional LSTM network, and the feature matrix is extracted hierarchically through the LSTM at the word and sentence level to obtain the fine-grained semantic representation. After that, the max-pooling layer converts the feature matrix into a lower-dimensional matrix, preserving only the obvious features. Finally, the feature matrix is taken as the input of a fully connected softmax layer, which contains a function that can convert the predicted linear vector into the output value as the probability of the text in each classification. Extensive experiments on two public benchmarks demonstrate the effectiveness of our proposed approach on an extremely small data set. It retains the state-of-the-art baseline performance in terms of precision, recall, accuracy, and F1 score, and through the model size, training time, and convergence epoch, we can conclude that our method can be deployed faster and lighter on mobile devices.