# Universal Language Model Fine-tuning for Text Classification

arXiv: https://arxiv.org/abs/1801.06146 · HTML: https://mlsanity.com/paper/1801.06146/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2018.

## Built on by (most cited first, 30 of 52)

- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [RoentGen: Vision-Language Foundation Model for Chest X-ray Generation](https://mlsanity.com/paper/2211.12737/) (2022)
- [How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks](https://mlsanity.com/paper/2303.00293/) (2023)
- [Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation](https://mlsanity.com/paper/2102.05766/) (2021)
- [Learning to Maximize Mutual Information for Chain-of-Thought Distillation](https://mlsanity.com/paper/2403.03348/) (2024)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [Person Re-Identification with a Locally Aware Transformer](https://mlsanity.com/paper/2106.03720/) (2021)
- [Mixed Distillation Helps Smaller Language Model Better Reasoning](https://mlsanity.com/paper/2312.10730/) (2023)
- [Attention-based Clinical Note Summarization](https://mlsanity.com/paper/2104.08942/) (2021)
- [Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity](https://mlsanity.com/paper/2112.14869/) (2021)
- [LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguation](https://mlsanity.com/paper/2104.08110/) (2021)
- [Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate](https://mlsanity.com/paper/2106.05903/) (2021)
- [Unlocking the Potential of Generative AI through Neuro-Symbolic Architectures: Benefits and Limitations](https://mlsanity.com/paper/2502.11269/) (2025)
- [NANOGPT: A Query-Driven Large Language Model Retrieval-Augmented Generation System for Nanotechnology Research](https://mlsanity.com/paper/2502.20541/) (2025)
- [Don't Think Twice! Over-Reasoning Impairs Confidence Calibration](https://mlsanity.com/paper/2508.15050/) (2025)
- [ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models](https://mlsanity.com/paper/2102.00238/) (2021)
- [Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction](https://mlsanity.com/paper/2207.02724/) (2022)
- [Best Practices for Large Language Models in Radiology](https://mlsanity.com/paper/2412.01233/) (2024)
- [Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design](https://mlsanity.com/paper/2305.13297/) (2023)
- [Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers](https://mlsanity.com/paper/2404.01317/) (2024)
- [Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge](https://mlsanity.com/paper/2404.01492/) (2024)
- [Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs](https://mlsanity.com/paper/2502.04357/) (2025)
- [Ceasing hate withMoH: Hate Speech Detection in Hindi-English Code-Switched Language](https://mlsanity.com/paper/2110.09393/) (2021)
- [Reaction-conditioned De Novo Enzyme Design with GENzyme](https://mlsanity.com/paper/2411.16694/) (2024)
- [Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction](https://mlsanity.com/paper/2412.13478/) (2024)
- [HindiLLM: Large Language Model for Hindi](https://mlsanity.com/paper/2412.20357/) (2024)
- [Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech](https://mlsanity.com/paper/2103.11792/) (2021)
- [Legal Search in Case Law and Statute Law](https://mlsanity.com/paper/2108.10127/) (2021)
- [Low Resource Summarization using Pre-trained Language Models](https://mlsanity.com/paper/2310.02790/) (2023)

## Abstract (arXiv)

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.