# A CLIP-Powered Framework for Robust and Generalizable Data Selection

arXiv: https://arxiv.org/abs/2410.11215 · HTML: https://mlsanity.com/paper/2410.11215/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 63 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Dataset Condensation with Differentiable Siamese Augmentation](https://mlsanity.com/paper/2102.08259/) (2021)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning](https://mlsanity.com/paper/2012.10630/) (2020)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)

## Built on by (most cited first, 21 of 21)

- [A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advances](https://mlsanity.com/paper/2505.17799/) (2025)
- [Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning](https://mlsanity.com/paper/2505.12212/) (2025)
- [EA-ViT: Efficient Adaptation for Elastic Vision Transformer](https://mlsanity.com/paper/2507.19360/) (2025)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning](https://mlsanity.com/paper/2602.11636/) (2026)
- [DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning](https://mlsanity.com/paper/2504.14810/) (2025)
- [Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors](https://mlsanity.com/paper/2509.21997/) (2025)
- [RS-Prune: Training-Free Data Pruning at High Ratios for Efficient Remote Sensing Diffusion Foundation Models](https://mlsanity.com/paper/2512.23239/) (2025)
- [RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment](https://mlsanity.com/paper/2506.21037/) (2025)
- [CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMs](https://mlsanity.com/paper/2510.18470/) (2025)
- [Position: Intelligent Science Laboratory Requires the Integration of Cognitive and Embodied AI](https://mlsanity.com/paper/2506.19613/) (2025)
- [ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models](https://mlsanity.com/paper/2509.15695/) (2025)
- [Rethinking Representativeness and Diversity in Dynamic Data Selection](https://mlsanity.com/paper/2603.04981/) (2026)
- [Data Agent: Learning to Select Data via End-to-End Dynamic Optimization](https://mlsanity.com/paper/2603.07433/) (2026)
- [Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning](https://mlsanity.com/paper/2507.12750/) (2025)
- [IPF-RDA: An Information-Preserving Framework for Robust Data Augmentation](https://mlsanity.com/paper/2509.16678/) (2025)
- [One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs](https://mlsanity.com/paper/2601.23041/) (2026)
- [Right Predictions, Misleading Explanations: On the Vulnerability of Vision-Language Model Explanations](https://mlsanity.com/paper/2605.16651/) (2026)
- [Delta-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation](https://mlsanity.com/paper/2508.09199/) (2025)
- [Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training](https://mlsanity.com/paper/2605.14773/) (2026)
- [Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training](https://mlsanity.com/paper/2610.12933/) (2026)

## Abstract (arXiv)

Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets invariably incurs substantial storage and computational overhead. Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples. To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. Specifically, our framework consists of three key modules-dataset adaptation, sample scoring, and selection optimization-that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality.