# Answering from Sure to Uncertain: Uncertainty-Aware Curriculum Learning for Video Question Answering

arXiv: https://arxiv.org/abs/2401.01510 · HTML: https://mlsanity.com/paper/2401.01510/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 75 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling](https://mlsanity.com/paper/2111.12681/) (2021)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [Revisiting the "Video" in Video-Language Understanding](https://mlsanity.com/paper/2206.01720/) (2022)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [Gradient Estimation Using Stochastic Computation Graphs](https://mlsanity.com/paper/1506.05254/) (2015)
- [Motion-Appearance Co-Memory Networks for Video Question Answering](https://mlsanity.com/paper/1803.10906/) (2018)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering](https://mlsanity.com/paper/2104.14085/) (2021)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding](https://mlsanity.com/paper/2103.16848/) (2021)
- [Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering](https://mlsanity.com/paper/2106.13432/) (2021)

## Built on by (most cited first, 2 of 2)

- [Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models](https://mlsanity.com/paper/2410.08174/) (2024)
- [Admitting Ignorance Helps the Video Question Answering Models to Answer](https://mlsanity.com/paper/2501.08771/) (2025)

## Abstract (arXiv)

While significant advancements have been made in video question answering (VideoQA), the potential benefits of enhancing model generalization through tailored difficulty scheduling have been largely overlooked in existing research. This paper seeks to bridge that gap by incorporating VideoQA into a curriculum learning (CL) framework that progressively trains models from simpler to more complex data. Recognizing that conventional self-paced CL methods rely on training loss for difficulty measurement, which might not accurately reflect the intricacies of video-question pairs, we introduce the concept of uncertainty-aware CL. Here, uncertainty serves as the guiding principle for dynamically adjusting the difficulty. Furthermore, we address the challenge posed by uncertainty by presenting a probabilistic modeling approach for VideoQA. Specifically, we conceptualize VideoQA as a stochastic computation graph, where the hidden representations are treated as stochastic variables. This yields two distinct types of uncertainty: one related to the inherent uncertainty in the data and another pertaining to the model's confidence. In practice, we seamlessly integrate the VideoQA model into our framework and conduct comprehensive experiments. The findings affirm that our approach not only achieves enhanced performance but also effectively quantifies uncertainty in the context of VideoQA.