# GhostEncoder: Stealthy Backdoor Attacks with Dynamic Triggers to Pre-trained Encoders in Self-supervised Learning

arXiv: https://arxiv.org/abs/2310.00626 · HTML: https://mlsanity.com/paper/2310.00626/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 1 Oct 2023.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [HiDDeN: Hiding Data With Deep Networks](https://mlsanity.com/paper/1807.09937/) (2018)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [Input-Aware Dynamic Backdoor Attack](https://mlsanity.com/paper/2010.08138/) (2020)
- [StegaStamp: Invisible Hyperlinks in Physical Photographs](https://mlsanity.com/paper/1904.05343/) (2019)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Poisoning and Backdooring Contrastive Learning](https://mlsanity.com/paper/2106.09667/) (2021)
- [Universal Litmus Patterns: Revealing Backdoor Attacks in CNNs](https://mlsanity.com/paper/1906.10842/) (2019)
- [BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning](https://mlsanity.com/paper/2108.00352/) (2021)
- [Backdoor Scanning for Deep Neural Networks through K-Arm Optimization](https://mlsanity.com/paper/2102.05123/) (2021)
- [Boosting Self-Supervised Learning via Knowledge Transfer](https://mlsanity.com/paper/1805.00385/) (2018)

## Built on by (most cited first, 6 of 6)

- [BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning](https://mlsanity.com/paper/2311.12075/) (2023)
- [Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models](https://mlsanity.com/paper/2502.18290/) (2025)
- [Backdooring Self-Supervised Contrastive Learning by Noisy Alignment](https://mlsanity.com/paper/2508.14015/) (2025)
- [SSL-OTA: Unveiling Backdoor Threats in Self-Supervised Learning for Object Detection](https://mlsanity.com/paper/2401.00137/) (2023)
- [Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models](https://mlsanity.com/paper/2512.00343/) (2025)
- [Repurposing and Evaluating the (In)Feasibility of Dataset Poisoning enabled Watermarking for Contrastive Learning](https://mlsanity.com/paper/2605.01834/) (2026)

## Abstract (arXiv)

Within the realm of computer vision, self-supervised learning (SSL) pertains to training pre-trained image encoders utilizing a substantial quantity of unlabeled images. Pre-trained image encoders can serve as feature extractors, facilitating the construction of downstream classifiers for various tasks. However, the use of SSL has led to an increase in security research related to various backdoor attacks. Currently, the trigger patterns used in backdoor attacks on SSL are mostly visible or static (sample-agnostic), making backdoors less covert and significantly affecting the attack performance. In this work, we propose GhostEncoder, the first dynamic invisible backdoor attack on SSL. Unlike existing backdoor attacks on SSL, which use visible or static trigger patterns, GhostEncoder utilizes image steganography techniques to encode hidden information into benign images and generate backdoor samples. We then fine-tune the pre-trained image encoder on a manipulation dataset to inject the backdoor, enabling downstream classifiers built upon the backdoored encoder to inherit the backdoor behavior for target downstream tasks. We evaluate GhostEncoder on three downstream tasks and results demonstrate that GhostEncoder provides practical stealthiness on images and deceives the victim model with a high attack success rate without compromising its utility. Furthermore, GhostEncoder withstands state-of-the-art defenses, including STRIP, STRIP-Cl, and SSL-Cleanse.