mlsanityData to 7 Oct 2026

Differentially Private Multimodal In-Context Learning

arXiv 2603.04894 ↗First posted 5 Mar 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision-language models are increasingly applied to sensitive domains such as medical imaging and personal photographs, yet existing differentially private methods for in-context learning are limited to few-shot, text-only settings because privacy cost scales with the number of tokens processed. We present Differentially Private Multimodal Task Vectors (DP-MTV), the first framework enabling many-shot multimodal in-context learning with formal (varepsilon, δ)-differential privacy by aggregating hundreds of demonstrations into compact task vectors in activation space. DP-MTV partitions private data into disjoint chunks, applies per-layer clipping to bound sensitivity, and adds calibrated noise to the aggregate, requiring only a single noise addition that enables unlimited inference queries. We evaluate on eight benchmarks across three VLM architectures, supporting deployment with or without auxiliary data. At varepsilon=1.0, DP-MTV achieves 50% on VizWiz compared to 55% non-private and 35% zero-shot, preserving most of the gain from in-context learning under meaningful privacy constraints.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 41 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

28 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety29 Aug
  2. 2024The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models2 Aug
  3. 2024Private Attribute Inference from Images with Vision-Language Models16 Apr

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  9. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  10. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  11. 2024What matters when building vision-language models?cited by 321
  12. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 216
13 more
  1. 2021Numerical Composition of Differential Privacycited by 174
  2. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  3. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  4. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  5. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  6. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  7. 2019Practical Differentially Private Top-k Selection with Pay-what-you-get Compositioncited by 38
  8. 2024Membership Inference Attacks Against In-Context Learningcited by 38
  9. 2024On the Privacy Risk of In-context Learningcited by 31
  10. 2024Granular Privacy Control for Geolocation with Vision Language Modelscited by 19
  11. 2024Locally Differentially Private In-Context Learningcited by 5
  12. 2025Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputscited by 2
  13. 2025Public Data Assisted Differentially Private In-Context Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
  2. 2026Differentially Private Paired Table-Image Multimodal Synthesis
  3. 2026Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs
  4. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning
  5. 2025ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
  6. 2026Unsafe2Safe: Controllable Image Anonymization for Downstream Utility
  7. 2025Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
  8. 2025Assessing Visual Privacy Risks in Multimodal AI: A Novel Taxonomy-Grounded Evaluation of Vision-Language Models