# STORM: Strategic Orchestration of Modalities for Rare Event Classification

arXiv: https://arxiv.org/abs/2412.02805 · HTML: https://mlsanity.com/paper/2412.02805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Builds on (most cited first, 11 of 11)

- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [Multimodal deep networks for text and image-based document classification](https://mlsanity.com/paper/1907.06370/) (2019)
- [EdGCon: Auto-assigner of Iconicity Ratings Grounded by Lexical Properties to Aid in Generation of Technical Gestures](https://mlsanity.com/paper/2306.01944/) (2023)
- [Merging Deep Learning with Expert Knowledge for Seizure Onset Zone localization from rs-fMRI in Pediatric Pharmaco Resistant Epilepsy](https://mlsanity.com/paper/2306.05572/) (2023)
- [CPS-LLM: Large Language Model based Safe Usage Plan Generator for Human-in-the-Loop Human-in-the-Plant Cyber-Physical System](https://mlsanity.com/paper/2405.11458/) (2024)
- [A simple method for measuring inequality](https://mlsanity.com/paper/2112.15284/) (2021)
- [Framework for developing and evaluating ethical collaboration between expert and machine](https://mlsanity.com/paper/2411.10983/) (2024)
- [High Fidelity Fast Simulation of Human in the Loop Human in the Plant (HIL-HIP) Systems](https://mlsanity.com/paper/2309.06558/) (2023)
- [The Expert Knowledge combined with AI outperforms AI Alone in Seizure Onset Zone Localization using resting state fMRI](https://mlsanity.com/paper/2312.09360/) (2023)
- [Transfer: Cross Modality Knowledge Transfer using Adversarial Networks -- A Study on Gesture Recognition](https://mlsanity.com/paper/2306.15114/) (2023)

## Built on by (most cited first, 2 of 2)

- [Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM](https://mlsanity.com/paper/2501.16481/) (2025)
- [Experience with Single Domain Generalization in Real World Medical Imaging Deployments](https://mlsanity.com/paper/2601.16359/) (2026)

## Abstract (arXiv)

In domains such as biomedical, expert insights are crucial for selecting the most informative modalities for artificial intelligence (AI) methodologies. However, using all available modalities poses challenges, particularly in determining the impact of each modality on performance and optimizing their combinations for accurate classification. Traditional approaches resort to manual trial and error methods, lacking systematic frameworks for discerning the most relevant modalities. Moreover, although multi-modal learning enables the integration of information from diverse sources, utilizing all available modalities is often impractical and unnecessary. To address this, we introduce an entropy-based algorithm STORM to solve the modality selection problem for rare event. This algorithm systematically evaluates the information content of individual modalities and their combinations, identifying the most discriminative features essential for rare class classification tasks. Through seizure onset zone detection case study, we demonstrate the efficacy of our algorithm in enhancing classification performance. By selecting useful subset of modalities, our approach paves the way for more efficient AI-driven biomedical analyses, thereby advancing disease diagnosis in clinical settings.