# A Comparison of LSTM and BERT for Small Corpus

arXiv: https://arxiv.org/abs/2009.05451 · HTML: https://mlsanity.com/paper/2009.05451/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2020.

## Built on by (most cited first, 30 of 40)

- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [TENT: Tensorized Encoder Transformer for Temperature Forecasting](https://mlsanity.com/paper/2106.14742/) (2021)
- [Personalized Benchmarking with the Ludwig Benchmarking Toolkit](https://mlsanity.com/paper/2111.04260/) (2021)
- [Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers](https://mlsanity.com/paper/2201.03425/) (2022)
- [Video Violence Recognition and Localization Using a Semi-Supervised Hard Attention Model](https://mlsanity.com/paper/2202.02212/) (2022)
- [Predicting Different Types of Subtle Toxicity in Unhealthy Online Conversations](https://mlsanity.com/paper/2106.03952/) (2021)
- [What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured](https://mlsanity.com/paper/2204.05673/) (2022)
- [Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate](https://mlsanity.com/paper/2310.14982/) (2023)
- [Do Language Models Understand Honorific Systems in Javanese?](https://mlsanity.com/paper/2502.20864/) (2025)
- [On the privacy-utility trade-off in differentially private hierarchical text classification](https://mlsanity.com/paper/2103.02895/) (2021)
- [Language Model-Based Paired Variational Autoencoders for Robotic Language Learning](https://mlsanity.com/paper/2201.06317/) (2022)
- [Discriminative and Generative Transformer-based Models For Situation Entity Classification](https://mlsanity.com/paper/2109.07434/) (2021)
- [Are fairness metric scores enough to assess discrimination biases in machine learning?](https://mlsanity.com/paper/2306.05307/) (2023)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [NLP4PBM: A Systematic Review on Process Extraction using Natural Language Processing with Rule-based, Machine and Deep Learning Methods](https://mlsanity.com/paper/2409.13738/) (2024)
- [Optimal Size-Performance Tradeoffs: Weighing PoS Tagger Models](https://mlsanity.com/paper/2104.07951/) (2021)
- [COVIDFakeExplainer: An Explainable Machine Learning based Web Application for Detecting COVID-19 Fake News](https://mlsanity.com/paper/2310.13890/) (2023)
- [Toward TransfORmers: Revolutionizing the Solution of Mixed Integer Programs with Transformers](https://mlsanity.com/paper/2402.13380/) (2024)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [DeepDiveAI: Identifying AI Related Documents in Large Scale Literature Data](https://mlsanity.com/paper/2408.12871/) (2024)
- [The effects of data size on Automated Essay Scoring engines](https://mlsanity.com/paper/2108.13275/) (2021)
- [A Twitter-Driven Deep Learning Mechanism for the Determination of Vehicle Hijacking Spots in Cities](https://mlsanity.com/paper/2208.10280/) (2022)
- [Human-mediated Large Language Models for Robotic Intervention in Children with Autism Spectrum Disorders](https://mlsanity.com/paper/2402.00260/) (2024)
- [Multi-objective Representation for Numbers in Clinical Narratives: A CamemBERT-Bio-Based Alternative to Large-Scale LLMs](https://mlsanity.com/paper/2405.18448/) (2024)
- [Multistep Brent Oil Price Forecasting with a Multi-Aspect Meta-heuristic Optimization and Ensemble Deep Learning Model](https://mlsanity.com/paper/2407.12062/) (2024)
- [FATE: Focal-modulated Attention Encoder for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2408.11336/) (2024)
- [Experimental Evaluation of Machine Learning Models for Goal-oriented Customer Service Chatbot with Pipeline Architecture](https://mlsanity.com/paper/2409.18568/) (2024)
- [EmoGRACE: Aspect-based emotion analysis for social media data](https://mlsanity.com/paper/2503.15133/) (2025)
- [Detecting AI Assistance in Abstract Complex Tasks](https://mlsanity.com/paper/2507.10761/) (2025)
- [Textual Analysis of Communications in COVID-19 Infected Community on Social Media](https://mlsanity.com/paper/2105.01189/) (2021)

## Abstract (arXiv)

Recent advancements in the NLP field showed that transfer learning helps with achieving state-of-the-art results for new tasks by tuning pre-trained models instead of starting from scratch. Transformers have made a significant improvement in creating new state-of-the-art results for many NLP tasks including but not limited to text classification, text generation, and sequence labeling. Most of these success stories were based on large datasets. In this paper we focus on a real-life scenario that scientists in academia and industry face frequently: given a small dataset, can we use a large pre-trained model like BERT and get better results than simple models? To answer this question, we use a small dataset for intent classification collected for building chatbots and compare the performance of a simple bidirectional LSTM model with a pre-trained BERT model. Our experimental results show that bidirectional LSTM models can achieve significantly higher results than a BERT model for a small dataset and these simple models get trained in much less time than tuning the pre-trained counterparts. We conclude that the performance of a model is dependent on the task and the data, and therefore before making a model choice, these factors should be taken into consideration instead of directly choosing the most popular model.