mlsanityData to 14 Oct 2026

Delta-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation

arXiv 2508.09199 ↗First posted 8 Aug 2025cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual Instruction Finetuning (VIF) is pivotal for post-training Vision-Language Models (VLMs). Unlike unimodal instruction finetuning in plain-text large language models, which mainly requires instruction datasets to enable model instruction-following ability, VIF also requires multimodal data to enable joint visual and textual understanding; therefore, it typically requires more data. Consequently, VIF imposes stricter data selection challenges: the method must scale efficiently to handle larger data demands while ensuring the quality of both visual and textual content, as well as their alignment. Despite its critical impact on performance, data selection for VIF remains an understudied area. In this paper, we propose Delta-AttnMask. This data-efficient framework quantifies sample quality through attention-guided masking of the model's hidden states, jointly evaluating image-text pairs without requiring domain labels, auxiliary models, or extra training. By computing loss differences (Delta) between the original states and states masked using high-attention regions, Delta-AttnMask intrinsically assesses sample quality. Experiments across multiple VLMs and datasets show that Delta-AttnMask achieves state-of-the-art performance with just 20% of data, accelerating training by 5x while surpassing full-dataset baselines by +10.1% in overall accuracy. Its model-agnostic and data-agnostic design ensures broad applicability across modalities and architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 31 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

21 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring10 Jun
  2. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks19 May
  3. 2025Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning10 Mar
  4. 2024ICONS: Influence Consensus for Vision-Language Data Selection31 Dec
  5. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness9 Dec
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  7. 2024Curriculum Learning with Quality-Driven Data Selection27 Jun
  8. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection19 Feb
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  5. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  7. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  8. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  9. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  10. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 21
  11. 2019Sensitivity Analysis of Deep Neural Networkscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Less is More: High-value Data Selection for Visual Instruction Tuning
  2. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  3. 2026Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning
  4. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection
  5. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
  6. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuning
  7. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning
  8. 2025Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents