# Ensemble Modeling for Multimodal Visual Action Recognition

arXiv: https://arxiv.org/abs/2308.05430 · HTML: https://mlsanity.com/paper/2308.05430/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Rethinking the Value of Labels for Improving Class-Imbalanced Learning](https://mlsanity.com/paper/2006.07529/) (2020)
- [CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning](https://mlsanity.com/paper/2102.09559/) (2021)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [M2m: Imbalanced Classification via Major-to-minor Translation](https://mlsanity.com/paper/2004.00431/) (2020)
- [Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification](https://mlsanity.com/paper/2103.14267/) (2021)
- [MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2103.12579/) (2021)
- [The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain](https://mlsanity.com/paper/2010.05654/) (2020)
- [Self Supervision to Distillation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2109.04075/) (2021)
- [RSG: A Simple but Effective Module for Learning Imbalanced Datasets](https://mlsanity.com/paper/2106.09859/) (2021)
- [Adjusting Decision Boundary for Class Imbalanced Learning](https://mlsanity.com/paper/1912.01857/) (2019)
- [On the effectiveness of task granularity for transfer learning](https://mlsanity.com/paper/1804.09235/) (2018)
- [Survey of resampling techniques for improving classification performance in unbalanced datasets](https://mlsanity.com/paper/1608.06048/) (2016)
- [Robust Face Recognition via Multimodal Deep Face Representation](https://mlsanity.com/paper/1509.00244/) (2015)
- [Mitigating Dataset Imbalance via Joint Generation and Classification](https://mlsanity.com/paper/2008.05524/) (2020)
- [Semi-supervised Long-tailed Recognition using Alternate Sampling](https://mlsanity.com/paper/2105.00133/) (2021)

## Built on by (most cited first, 1 of 1)

- [G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding](https://mlsanity.com/paper/2608.20157/) (2026)

## Abstract (arXiv)

In this work, we propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset. Based on the underlying principle of focal loss, which captures the relationship between tail (scarce) classes and their prediction difficulties, we propose an exponentially decaying variant of focal loss for our current task. It initially emphasizes learning from the hard misclassified examples and gradually adapts to the entire range of examples in the dataset. This annealing process encourages the model to strike a balance between focusing on the sparse set of hard samples, while still leveraging the information provided by the easier ones. Additionally, we opt for the late fusion strategy to combine the resultant probability distributions from RGB and Depth modalities for final action prediction. Experimental evaluations on the MECCANO dataset demonstrate the effectiveness of our approach.