mlsanityData to 14 Oct 2026

Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models

arXiv 2311.01441 ↗First posted 2 Nov 2023cs.LG cs.AI cs.CVcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

We propose a conceptually simple and lightweight framework for improving the robustness of vision models through the combination of knowledge distillation and data augmentation. We address the conjecture that larger models do not make for better teachers by showing strong gains in out-of-distribution robustness when distilling from pretrained foundation models. Following this finding, we propose Discrete Adversarial Distillation (DAD), which leverages a robust teacher to generate adversarial examples and a VQGAN to discretize them, creating more informative samples than standard data augmentation techniques. We provide a theoretical framework for the use of a robust teacher in the knowledge distillation with data augmentation setting and demonstrate strong gains in out-of-distribution robustness and clean accuracy across different student architectures. Notably, our method adds minor computational overhead compared to similar techniques and can be easily combined with other data augmentations for further improvements.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 84 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

73 of 84 references.

References · 73

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2013Intriguing properties of neural networkscited by 5,155
  11. 2017Neural Discrete Representation Learningcited by 4,712
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  12. 2019Natural Adversarial Examplescited by 1,408
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  14. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  15. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  16. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  17. 2019Relational Knowledge Distillationcited by 889
  18. 2018Robustness May Be at Odds with Accuracycited by 880
  19. 2019Contrastive Representation Distillationcited by 748
  20. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  21. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  22. 2019Similarity-Preserving Knowledge Distillationcited by 516
  23. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  24. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  25. 2019On the Efficacy of Knowledge Distillationcited by 381
  26. 2018Adversarially Robust Generalization Requires More Datacited by 318
  27. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  28. 2019Adversarial Examples Improve Image Recognitioncited by 286
  29. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  30. 2021Improving Robustness using Generated Datacited by 234
  31. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  32. 2022Knowledge Distillation from A Stronger Teachercited by 208
  33. 2020Understanding and Mitigating the Tradeoff Between Robustness and Accuracycited by 149
  34. 2019Adversarial Robustness through Local Linearizationcited by 146
  35. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
  36. 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
  37. 2019Adversarially Robust Distillationcited by 125
  38. 2019Adversarial AutoAugmentcited by 111

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2024Rethinking model prototyping through the MedMNIST+ dataset collectioncited by 9
  2. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  3. 2025OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triadcited by 6
  4. 2024Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Datacited by 4
  5. 2025Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Modelscited by 3
  6. 2024TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Abilitycited by 2
  7. 2025Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentationcited by 2
  8. 2024Domain Generalization Guided by Large-Scale Pre-Trained Priorscited by 1
  9. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Models
  2. 2025Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation
  3. 2026Learning Better Certified Models from Empirically-Robust Teachers
  4. 2022ARDIR: Improving Robustness using Knowledge Distillation of Internal Representation
  5. 2022On the benefits of knowledge distillation for adversarial robustness
  6. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  7. 2022Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training
  8. 2023HARD: Hard Augmentations for Robust Distillation