# Curriculum Learning for Data-Efficient Vision-Language Alignment

arXiv: https://arxiv.org/abs/2207.14525 · HTML: https://mlsanity.com/paper/2207.14525/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Contrastive Learning for Weakly Supervised Phrase Grounding](https://mlsanity.com/paper/2006.09920/) (2020)
- [Does Vision-and-Language Pretraining Improve Lexical Grounding?](https://mlsanity.com/paper/2109.10246/) (2021)

## Built on by (most cited first, 11 of 11)

- [UnIVAL: Unified Model for Image, Video, Audio and Language Tasks](https://mlsanity.com/paper/2307.16184/) (2023)
- [Robust Domain Misinformation Detection via Multi-modal Feature Alignment](https://mlsanity.com/paper/2311.14315/) (2023)
- [Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?](https://mlsanity.com/paper/2409.17080/) (2024)
- [Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training](https://mlsanity.com/paper/2410.15509/) (2024)
- [Learning from Children: Improving Image-Caption Pretraining via Curriculum](https://mlsanity.com/paper/2305.17540/) (2023)
- [Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIP](https://mlsanity.com/paper/2601.09859/) (2026)
- [Adapting Lightweight Vision Language Models for Radiological Visual Question Answering](https://mlsanity.com/paper/2506.14451/) (2025)
- [When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA](https://mlsanity.com/paper/2511.17886/) (2025)
- [MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration](https://mlsanity.com/paper/2603.09101/) (2026)
- [Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method](https://mlsanity.com/paper/2603.27556/) (2026)
- [Soft Curriculum Learning for Optimizing Fresh and Generalized Recommendations](https://mlsanity.com/paper/2609.35783/) (2026)

## Abstract (arXiv)

Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all image-text pairs contain the same object to learn finer-grained contextual alignment. Aligning pre-trained BERT and VinVL models to each other using TOnICS outperforms CLIP on downstream zero-shot image retrieval while using less than 1% as much training data.