mlsanityData to 14 Oct 2026

Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training

arXiv 2608.16927 ↗First posted 5 Aug 2026cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

As supervised fine-tuning data continues to scale, selecting high-value subsets from large candidate pools is crucial for reducing training cost and improving model performance. Existing methods often measure diversity directly in the original embedding space, where geometric metrics entangle dominant semantic directions, fine-grained supervision differences, and local noise. We address this limitation by formulating data selection as a coarse-to-fine hierarchical coverage problem and propose MASS. MASS learns low-dimensional principal manifold coordinates with a dense autoencoder for coarse semantic grouping, and then performs quality-aware sparse feature coverage within each group using a TopK sparse autoencoder. Experiments on Vision Flan and LLaVA-CoT show that MASS consistently outperforms strong data selection baselines across multiple budgets, and in several settings matches or surpasses full data training with only a small subset of data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

33 of 50 references.

References · 33

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2016A Diagram Is Worth A Dozen Imagescited by 804
21 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  3. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  4. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  5. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  6. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  7. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  8. 2013k-Sparse Autoencoderscited by 215
  9. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  10. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuningcited by 73
  11. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  12. 2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 61
  13. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  14. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 29
  15. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  16. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 15
  17. 2025MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Spacecited by 12
  18. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  19. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  20. 2026ScalSelect: Scalable Training-Free Multimodal Data Selection for Efficient Visual Instruction Tuningcited by 5
  21. 2026Does the Question Really Matter? Training-Free Data Selection for Vision-Language SFTcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation
  2. 2025Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning
  3. 2023Score-based generative models learn manifold-like structures with constrained mixing
  4. 2026Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training
  5. 2026Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders
  6. 2024How to Train Data-Efficient LLMs
  7. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selection
  8. 2025Meta-Learning and Synthetic Data for Automated Pretraining and Finetuning