# ReToMe-VA: Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack

arXiv: https://arxiv.org/abs/2408.05479 · HTML: https://mlsanity.com/paper/2408.05479/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 36 of its 49 references that are in the map.
- First posted on arXiv 10 Aug 2024.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Countering Adversarial Images using Input Transformations](https://mlsanity.com/paper/1711.00117/) (2017)
- [Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks](https://mlsanity.com/paper/1704.01155/) (2017)
- [Diffusion Models for Adversarial Purification](https://mlsanity.com/paper/2205.07460/) (2022)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser](https://mlsanity.com/paper/1712.02976/) (2017)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [ColorFool: Semantic Adversarial Colorization](https://mlsanity.com/paper/1911.10891/) (2019)
- [Semantic Adversarial Examples](https://mlsanity.com/paper/1804.00499/) (2018)
- [Unrestricted Adversarial Examples via Semantic Manipulation](https://mlsanity.com/paper/1904.06347/) (2019)
- [Functional Adversarial Attacks](https://mlsanity.com/paper/1906.00001/) (2019)
- [Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distance](https://mlsanity.com/paper/1911.02466/) (2019)
- [Content-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2305.10665/) (2023)

## Built on by (most cited first, 3 of 3)

- [Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos](https://mlsanity.com/paper/2504.14921/) (2025)
- [CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMs](https://mlsanity.com/paper/2507.00817/) (2025)
- [Embedding Hidden Adversarial Capabilities in Pre-Trained Diffusion Models](https://mlsanity.com/paper/2504.08782/) (2025)

## Abstract (arXiv)

Recent diffusion-based unrestricted attacks generate imperceptible adversarial examples with high transferability compared to previous unrestricted attacks and restricted attacks. However, existing works on diffusion-based unrestricted attacks are mostly focused on images yet are seldom explored in videos. In this paper, we propose the Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack (ReToMe-VA), which is the first framework to generate imperceptible adversarial video clips with higher transferability. Specifically, to achieve spatial imperceptibility, ReToMe-VA adopts a Timestep-wise Adversarial Latent Optimization (TALO) strategy that optimizes perturbations in diffusion models' latent space at each denoising step. TALO offers iterative and accurate updates to generate more powerful adversarial frames. TALO can further reduce memory consumption in gradient computation. Moreover, to achieve temporal imperceptibility, ReToMe-VA introduces a Recursive Token Merging (ReToMe) mechanism by matching and merging tokens across video frames in the self-attention module, resulting in temporally consistent adversarial videos. ReToMe concurrently facilitates inter-frame interactions into the attack process, inducing more diverse and robust gradients, thus leading to better adversarial transferability. Extensive experiments demonstrate the efficacy of ReToMe-VA, particularly in surpassing state-of-the-art attacks in adversarial transferability by more than 14.16% on average.