# Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities

arXiv: https://arxiv.org/abs/2508.06800 · HTML: https://mlsanity.com/paper/2508.06800/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 9 Aug 2025.

## Builds on (most cited first, 23 of 23)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Competence-based Curriculum Learning for Neural Machine Translation](https://mlsanity.com/paper/1903.09848/) (2019)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation](https://mlsanity.com/paper/1509.03150/) (2015)
- [Deep Partial Multi-View Learning](https://mlsanity.com/paper/2011.06170/) (2020)
- [Retrieval-Augmented Generation for Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.13193/) (2024)
- [Hard Patches Mining for Masked Image Modeling](https://mlsanity.com/paper/2304.05919/) (2023)
- [Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion](https://mlsanity.com/paper/1911.07848/) (2019)
- [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones](https://mlsanity.com/paper/2211.09703/) (2022)
- [Towards Human-Machine Cooperation: Self-supervised Sample Mining for Object Detection](https://mlsanity.com/paper/1803.09867/) (2018)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities](https://mlsanity.com/paper/2210.15359/) (2022)
- [Deep Instance-Level Hard Negative Mining Model for Histopathology Images](https://mlsanity.com/paper/1906.09681/) (2019)
- [M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentation](https://mlsanity.com/paper/2205.02524/) (2022)
- [M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2310.14605/) (2023)
- [EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2410.13179/) (2024)

## Built on by (most cited first, 2 of 2)

- [Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective](https://mlsanity.com/paper/2608.30563/) (2026)
- [Buffer replay enhances the robustness of multimodal learning under missing-modality](https://mlsanity.com/paper/2511.23070/) (2025)

## Abstract (arXiv)

Missing modalities have recently emerged as a critical research direction in multimodal emotion recognition (MER). Conventional approaches typically address this issue through missing modality reconstruction. However, these methods fail to account for variations in reconstruction difficulty across different samples, consequently limiting the model's ability to handle hard samples effectively. To overcome this limitation, we propose a novel Hardness-Aware Dynamic Curriculum Learning framework, termed HARDY-MER. Our framework operates in two key stages: first, it estimates the hardness level of each sample, and second, it strategically emphasizes hard samples during training to enhance model performance on these challenging instances. Specifically, we first introduce a Multi-view Hardness Evaluation mechanism that quantifies reconstruction difficulty by considering both Direct Hardness (modality reconstruction errors) and Indirect Hardness (cross-modal mutual information). Meanwhile, we introduce a Retrieval-based Dynamic Curriculum Learning strategy that dynamically adjusts the training curriculum by retrieving samples with similar semantic information and balancing the learning focus between easy and hard instances. Extensive experiments on benchmark datasets demonstrate that HARDY-MER consistently outperforms existing methods in missing-modality scenarios. Our code will be made publicly available at https://github.com/HARDY-MER/HARDY-MER.