mlsanityData to 14 Oct 2026

PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

arXiv 2502.12119 ↗First posted 17 Feb 2025cs.CV cs.AI cs.CLcited by 61

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a textitGlobal Semantic Drift, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise PRISM, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7% relative improvement over the baseline. The code is available for access via hrefhttps://github.com/bibisbar/PRISMthis repository.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 64 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

32 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024ICONS: Influence Consensus for Vision-Language Data Selection31 Dec
  2. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering16 Dec
  3. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness9 Dec
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024Wings: Learning Multimodal LLMs without Text-only Forgetting5 Jun
  8. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection19 Feb
  9. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023LIMA: Less Is More for Alignmentcited by 1,100
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
8 more
  1. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  5. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  6. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  7. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  8. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 32; 2026: 29.

61 papers.

Sort:

By area:

  1. 2026ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrievalcited by 41
  2. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLMcited by 37
  3. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
  4. 2026INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrievalcited by 26
  5. 2026HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrievalcited by 24
  6. 2025Self-Evolving Multi-Agent Systems via Textual Backpropagationcited by 23
  7. 2026MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolutioncited by 22
  8. 2025MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answeringcited by 19
  9. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  10. 2026Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrievalcited by 18
  11. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  12. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
38 more
  1. 2026Graph is a Substrate Across Data Modalitiescited by 14
  2. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 13
  3. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  4. 2025GeometryZero: Advancing Geometry Solving via Group Contrastive Policy Optimizationcited by 9
  5. 2025SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysiscited by 9
  6. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 8
  7. 2026What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generationcited by 8
  8. 2026OPD-V: Visual On-Policy Self-Distillation with Modality Balancecited by 8
  9. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 7
  10. 2025MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Modelscited by 7
  11. 2025Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasetscited by 6
  12. 2025TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Taskscited by 6
  13. 2026ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoningcited by 6
  14. 2025PreP-OCR: A Complete Pipeline for Document Image Restoration and Enhanced OCR Accuracycited by 5
  15. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understandingcited by 5
  16. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  17. 2025A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planningcited by 4
  18. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Mediacited by 4
  19. 2025From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generationcited by 4
  20. 2026Set-Valued Prediction for Large Language Models with Feasibility-Aware Coverage Guaranteescited by 4
  21. 2025Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Predictioncited by 3
  22. 2025Differentiable NMS via Sinkhorn Matching for End-to-End Fabric Defect Detectioncited by 3
  23. 2025AStF: Motion Style Transfer via Adaptive Statistics Fusorcited by 3
  24. 2025Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarkscited by 3
  25. 2026TCAP: Tri-Component Attention Profiling for Unsupervised Backdoor Detection in MLLM Fine-Tuningcited by 3
  26. 2025TRACE: Trajectory-Constrained Concept Erasure in Diffusion Modelscited by 2
  27. 2025Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guaranteescited by 2
  28. 2026IndusAgent: Reinforcing Open-Vocabulary Industrial Anomaly Detection with Agentic Toolscited by 2
  29. 2025Concepts or Skills? Rethinking Instruction Selection for Multi-modal Modelscited by 1
  30. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1
  31. 2026Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuningcited by 1
  32. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guaranteescited by 1
  33. 2025LLMSR@XLLM25: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillationcited by 0
  34. 2025(Almost) Free Modality Stitching of Foundation Modelscited by 0
  35. 2025FADE: Adversarial Concept Erasure in Flow Modelscited by 0
  36. 2025Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Controlcited by 0
  37. 2025Robust Concept Erasure in Diffusion Models: A Theoretical Perspective on Security and Robustnesscited by 0
  38. 2026When Denoising Becomes Unsigning: Theoretical and Empirical Analysis of Watermark Fragility Under Diffusion-Based Image Editingcited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  2. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization
  3. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
  4. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Models
  5. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories
  6. 2026MAGIC: Multimodal Alignment & Grounding-aware Instruction Coreset for Vision-Language Models
  7. 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models
  8. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning