# Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks

arXiv: https://arxiv.org/abs/1611.06204 · HTML: https://mlsanity.com/paper/1611.06204/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2016.

## Built on by (most cited first, 30 of 44)

- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [LeanAgent: Lifelong Learning for Formal Theorem Proving](https://mlsanity.com/paper/2410.06209/) (2024)
- [Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning](https://mlsanity.com/paper/2506.11300/) (2025)
- [Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent](https://mlsanity.com/paper/2106.06080/) (2021)
- [RealCustom++: Representing Images as Real Textual Word for Real-Time Customization](https://mlsanity.com/paper/2408.09744/) (2024)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [Self-Guided Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/2105.04475/) (2021)
- [Learning Rate Curriculum](https://mlsanity.com/paper/2205.09180/) (2022)
- [Too Much Information: Keeping Training Simple for BabyLMs](https://mlsanity.com/paper/2311.01955/) (2023)
- [Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems](https://mlsanity.com/paper/2309.15995/) (2023)
- [A Psychology-based Unified Dynamic Framework for Curriculum Learning](https://mlsanity.com/paper/2408.05326/) (2024)
- [DiscrimLoss: A Universal Loss for Hard Samples and Incorrect Samples Discrimination](https://mlsanity.com/paper/2208.09884/) (2022)
- [CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family](https://mlsanity.com/paper/2210.00313/) (2022)
- [Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU](https://mlsanity.com/paper/2210.12499/) (2022)
- [Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach](https://mlsanity.com/paper/2402.14948/) (2024)
- [A Quality-Centric Framework for Generic Deepfake Detection](https://mlsanity.com/paper/2411.05335/) (2024)
- [CBM: Curriculum by Masking](https://mlsanity.com/paper/2407.05193/) (2024)
- [Demystifying Data Organization for Enhanced LLM Training](https://mlsanity.com/paper/2605.30334/) (2026)
- [Analyzing Curriculum Learning for Sentiment Analysis along Task Difficulty, Pacing and Visualization Axes](https://mlsanity.com/paper/2102.09990/) (2021)
- [Self-supervised Adaptive Pre-training of Multilingual Speech Models for Language and Dialect Identification](https://mlsanity.com/paper/2312.07338/) (2023)
- [Bridging Social Media and Search Engines: Dredge Words and the Detection of Unreliable Domains](https://mlsanity.com/paper/2406.11423/) (2024)
- [Unified Long-Term Time-Series Forecasting Benchmark](https://mlsanity.com/paper/2309.15946/) (2023)
- [STCL:Curriculum learning Strategies for deep learning image steganography models](https://mlsanity.com/paper/2504.17609/) (2025)
- [Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition](https://mlsanity.com/paper/2106.04392/) (2021)
- [A Syntax Aware BERT for Identifying Well-Formed Queries in a Curriculum Framework](https://mlsanity.com/paper/2208.09912/) (2022)
- [Unveiling the frontiers of deep learning: innovations shaping diverse domains](https://mlsanity.com/paper/2309.02712/) (2023)
- [Enhancing Multilingual Speech Recognition through Language Prompt Tuning and Frame-Level Language Adapter](https://mlsanity.com/paper/2309.09443/) (2023)
- [GMTR: Graph Matching Transformers](https://mlsanity.com/paper/2311.08141/) (2023)

## Abstract (arXiv)

Curriculum Learning emphasizes the order of training instances in a computational learning setup. The core hypothesis is that simpler instances should be learned early as building blocks to learn more complex ones. Despite its usefulness, it is still unknown how exactly the internal representation of models are affected by curriculum learning. In this paper, we study the effect of curriculum learning on Long Short-Term Memory (LSTM) networks, which have shown strong competency in many Natural Language Processing (NLP) problems. Our experiments on sentiment analysis task and a synthetic task similar to sequence prediction tasks in NLP show that curriculum learning has a positive effect on the LSTM's internal states by biasing the model towards building constructive representations i.e. the internal representation at the previous timesteps are used as building blocks for the final prediction. We also find that smaller models significantly improves when they are trained with curriculum learning. Lastly, we show that curriculum learning helps more when the amount of training data is limited.