# CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models

arXiv: https://arxiv.org/abs/2507.22828 · HTML: https://mlsanity.com/paper/2507.22828/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts](https://mlsanity.com/paper/2311.05608/) (2023)
- [EfficientFormer: Vision Transformers at MobileNet Speed](https://mlsanity.com/paper/2206.01191/) (2022)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Improving Multimodal Datasets with Image Captioning](https://mlsanity.com/paper/2307.10350/) (2023)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [A Stealthy Wrongdoer: Feature-Oriented Reconstruction Attack against Split Learning](https://mlsanity.com/paper/2405.04115/) (2024)
- [SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization](https://mlsanity.com/paper/2410.10759/) (2024)
- [Passive Inference Attacks on Split Learning via Adversarial Regularization](https://mlsanity.com/paper/2310.10483/) (2023)
- [Secure Federated XGBoost with CUDA-accelerated Homomorphic Encryption via NVIDIA FLARE](https://mlsanity.com/paper/2504.03909/) (2025)

## Built on by (most cited first, 3 of 3)

- [LeakyCLIP: Extracting Training Data from CLIP](https://mlsanity.com/paper/2508.00756/) (2025)
- [Position: Life-Logging Video Streams Make the Privacy-Utility Trade-off Inevitable](https://mlsanity.com/paper/2605.10404/) (2026)
- [TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction](https://mlsanity.com/paper/2607.04484/) (2026)

## Abstract (arXiv)

As Vision-Language Models (VLMs) are increasingly deployed in split-DNN configurations--with visual encoders (e.g., ResNet, ViT) operating on user devices and sending intermediate features to the cloud--there is a growing privacy risk from semantic information leakage. Existing approaches to reconstructing images from these intermediate features often result in blurry, semantically ambiguous images. To directly address semantic leakage, we propose CapRecover, a cross-modality inversion framework that recovers high-level semantic content, such as labels or captions, directly from intermediate features without image reconstruction. We evaluate CapRecover on multiple datasets and victim models, demonstrating strong performance in semantic recovery. Specifically, CapRecover achieves up to 92.71% Top-1 label accuracy on CIFAR-10 and generates fluent captions from ResNet50 features on COCO2017 with ROUGE-L scores up to 0.52. Our analysis further reveals that deeper convolutional layers encode significantly more semantic information compared to shallow layers. To mitigate semantic leakage, we introduce a simple yet effective protection method: adding random noise to intermediate features at each layer and removing the noise in the next layer. Experimental results show that this approach prevents semantic leakage without additional training costs. Our code is available at https://jus1mple.github.io/Image2CaptionAttack.