mlsanityData to 13 Oct 2026

Low-Rank Similarity Mining for Multimodal Dataset Distillation

arXiv 2406.03793 ↗First posted 6 Jun 2024cs.LG cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Though dataset distillation has witnessed rapid development in recent years, the distillation of multimodal data, e.g., image-text pairs, poses unique and under-explored challenges. Unlike unimodal data, image-text contrastive learning (ITC) data lack inherent categorization and should instead place greater emphasis on modality correspondence. In this work, we propose Low-Rank Similarity Mining (LoRS) for multimodal dataset distillation, that concurrently distills a ground truth similarity matrix with image-text pairs, and leverages low-rank factorization for efficiency and scalability. The proposed approach brings significant improvement to the existing algorithms, marking a significant contribution to the field of visual-language dataset distillation. We advocate adopting LoRS as a foundational synthetic data setup for image-text dataset distillation. Our code is available at https://github.com/silicx/LoRS_Distill.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

49 of 68 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2023Visual Instruction Tuningcited by 7,827
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,282
  11. 2013Describing Textures in the Wildcited by 2,115
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
37 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  3. 2020Designing Network Design Spacescited by 956
  4. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 675
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 648
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 639
  7. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  8. 2020Dataset Condensation with Gradient Matchingcited by 512
  9. 2022Dataset Distillation by Matching Training Trajectoriescited by 438
  10. 2021Dataset Condensation with Distribution Matchingcited by 363
  11. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  12. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  13. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  14. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 216
  15. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  16. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  17. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  18. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 137
  19. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  20. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  21. 2022Dataset Distillation via Factorizationcited by 115
  22. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  23. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  24. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  25. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  26. 2022Synthesizing Informative Training Samples with GANcited by 77
  27. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  28. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Strongercited by 56
  29. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  30. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  31. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34
  32. 2023Frequency Domain-based Dataset Distillationcited by 28
  33. 2021Data Distillation for Text Classificationcited by 25
  34. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  35. 2023CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Losscited by 12
  36. 2022On Divergence Measures for Bayesian Pseudocoresetscited by 10
  37. 2023Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillationcited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 25
  2. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  3. 2025Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillationcited by 6
  4. 2025Efficient Multimodal Dataset Distillation via Generative Modelscited by 6
  5. 2025ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensationcited by 4
  6. 2026Multimodal Distribution Matching for Vision-Language Dataset Distillationcited by 3
  7. 2025CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encodercited by 2
  8. 2025Multi-Modal Dataset Distillation in the Wildcited by 1
  9. 2026Multimodal Dataset Distillation via Phased Teacher Modelscited by 1
  10. 2025Leveraging Multi-Modal Information to Enhance Dataset Distillationcited by 0
  11. 2025Technical Report on Text Dataset Distillationcited by 0
  12. 2025Distill Video Datasets into Imagescited by 0
4 more
  1. 2026Multimodal Dataset Distillation Made Simple by Prototype-Guided Data Synthesiscited by 0
  2. 2026Omnimodal Dataset Distillation via High-order Proxy Alignmentcited by 0
  3. 2026CAST: Collapse-Aware multi-Scale Topology Fusion for Multimodal Coreset Selectioncited by 0
  4. 2026Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillationcited by 0