# Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training

arXiv: https://arxiv.org/abs/2410.15509 · HTML: https://mlsanity.com/paper/2410.15509/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Power Hungry Processing: Watts Driving the Cost of AI Deployment?](https://mlsanity.com/paper/2311.16863/) (2023)
- [When Do Curricula Work?](https://mlsanity.com/paper/2012.03107/) (2020)
- [Dynamic Curriculum Learning for Imbalanced Data Classification](https://mlsanity.com/paper/1901.06783/) (2019)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021)
- [[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2404.06214/) (2024)
- [CLIMB: Curriculum Learning for Infant-inspired Model Building](https://mlsanity.com/paper/2311.08886/) (2023)
- [A Competence-aware Curriculum for Visual Concepts Learning via Question Answering](https://mlsanity.com/paper/2007.01499/) (2020)
- [Curriculum Learning for Data-Efficient Vision-Language Alignment](https://mlsanity.com/paper/2207.14525/) (2022)
- [Curriculum Learning with Diversity for Supervised Computer Vision Tasks](https://mlsanity.com/paper/2009.10625/) (2020)
- [Too Much Information: Keeping Training Simple for BabyLMs](https://mlsanity.com/paper/2311.01955/) (2023)
- [Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?](https://mlsanity.com/paper/2311.18761/) (2023)
- [Learning from Children: Improving Image-Caption Pretraining via Curriculum](https://mlsanity.com/paper/2305.17540/) (2023)

## Built on by (most cited first, 3 of 3)

- [Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling](https://mlsanity.com/paper/2510.08470/) (2025)
- [Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models](https://mlsanity.com/paper/2510.01845/) (2025)
- [When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA](https://mlsanity.com/paper/2511.17886/) (2025)

## Abstract (arXiv)

For specialized domains, there is often not a wealth of data with which to train large machine learning models. In such limited data / compute settings, various methods exist aiming to textitdo more with less, such as finetuning from a pretrained model, modulating difficulty levels as data are presented to a model (curriculum learning), and considering the role of model type / size. Approaches to efficient textitmachine learning also take inspiration from textithuman learning by considering use cases where machine learning systems have access to approximately the same number of words experienced by a 13 year old child (100M words). We investigate the role of 3 primary variables in a limited data regime as part of the multimodal track of the BabyLM challenge. We contrast: (i) curriculum learning, (ii), pretraining (with text-only data), (iii) model type. We modulate these variables and assess them on two types of tasks: (a) multimodal (text+image), and (b) unimodal (text-only) tasks. We find that curriculum learning benefits multimodal evaluations over non-curriclum learning models, particularly when combining text-only pretraining. On text-only tasks, curriculum learning appears to help models with smaller trainable parameter counts. We suggest possible reasons based on architectural differences and training designs as to why one might observe such results.