# SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilation

arXiv: https://arxiv.org/abs/2409.13860 · HTML: https://mlsanity.com/paper/2409.13860/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements](https://mlsanity.com/paper/2401.06766/) (2024)
- [Active Learning for Deep Object Detection via Probabilistic Modeling](https://mlsanity.com/paper/2103.16130/) (2021)
- [Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic](https://mlsanity.com/paper/2404.07177/) (2024)
- [Data Pruning via Moving-one-Sample-out](https://mlsanity.com/paper/2310.14664/) (2023)
- [Scalable Active Learning for Object Detection](https://mlsanity.com/paper/2004.04699/) (2020)
- [Localization-Aware Active Learning for Object Detection](https://mlsanity.com/paper/1801.05124/) (2018)
- [Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification](https://mlsanity.com/paper/1910.08168/) (2019)
- [FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication](https://mlsanity.com/paper/2404.16123/) (2024)
- [NVAutoNet: Fast and Accurate 360circ 3D Visual Perception For Self Driving](https://mlsanity.com/paper/2303.12976/) (2023)
- [Thinking Like an Annotator: Generation of Dataset Labeling Instructions](https://mlsanity.com/paper/2306.14035/) (2023)

## Built on by (most cited first, 7 of 7)

- [AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training](https://mlsanity.com/paper/2507.00049/) (2025)
- [Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development](https://mlsanity.com/paper/2607.20532/) (2026)
- [Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems](https://mlsanity.com/paper/2604.08366/) (2026)
- [Multimodal Cinematic Video Synthesis Using Text-to-Image and Audio Generation Models](https://mlsanity.com/paper/2506.10005/) (2025)
- [From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence](https://mlsanity.com/paper/2607.16560/) (2026)
- [Test-Time Coverage: Test-Conditioned Data Curation for Deployment-Aware Learning](https://mlsanity.com/paper/2607.22697/) (2026)
- [IronViT: Toward Efficient Generalist Visual Representation Learning](https://mlsanity.com/paper/2609.29252/) (2026)

## Abstract (arXiv)

In recent years, the data collected for artificial intelligence has grown to an unmanageable amount. Particularly within industrial applications, such as autonomous vehicles, model training computation budgets are being exceeded while model performance is saturating -- and yet more data continues to pour in. To navigate the flood of data, we propose a framework to select the most semantically diverse and important dataset portion. Then, we further semantically enrich it by discovering meaningful new data from a massive unlabeled data pool. Importantly, we can provide explainability by leveraging foundation models to generate semantics for every data point. We quantitatively show that our Semantic Selection and Enrichment framework (SSE) can a) successfully maintain model performance with a smaller training dataset and b) improve model performance by enriching the smaller dataset without exceeding the original dataset size. Consequently, we demonstrate that semantic diversity is imperative for optimal data selection and model performance.