# Automated Side Channel Analysis of Media Software with Manifold Learning

arXiv: https://arxiv.org/abs/2112.04947 · HTML: https://mlsanity.com/paper/2112.04947/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 119 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Image reconstruction by domain transform manifold learning](https://mlsanity.com/paper/1704.08841/) (2017)
- [Deep Cross-Modal Audio-Visual Generation](https://mlsanity.com/paper/1704.08292/) (2017)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [SphereReID: Deep Hypersphere Manifold Embedding for Person Re-Identification](https://mlsanity.com/paper/1807.00537/) (2018)
- [NEUZZ: Efficient Fuzzing with Neural Program Smoothing](https://mlsanity.com/paper/1807.05620/) (2018)
- [Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks](https://mlsanity.com/paper/1709.08041/) (2017)
- [Software Grand Exposure: SGX Cache Attacks Are Practical](https://mlsanity.com/paper/1702.07521/) (2017)
- [Cascade Attention Guided Residue Learning GAN for Cross-Modal Translation](https://mlsanity.com/paper/1907.01826/) (2019)
- [Deep Long Audio Inpainting](https://mlsanity.com/paper/1911.06476/) (2019)
- [Malware Guard Extension: Using SGX to Conceal Cache Attacks](https://mlsanity.com/paper/1702.08719/) (2017)
- [CacheZoom: How SGX Amplifies The Power of Cache Attacks](https://mlsanity.com/paper/1703.06986/) (2017)
- [Neutaint: Efficient Dynamic Taint Analysis with Neural Networks](https://mlsanity.com/paper/1907.03756/) (2019)
- [An Off-Chip Attack on Hardware Enclaves via the Memory Bus](https://mlsanity.com/paper/1912.01701/) (2019)
- [The Spy in the Sandbox -- Practical Cache Attacks in Javascript](https://mlsanity.com/paper/1502.07373/) (2015)
- [Unsupervised Latent Behavior Manifold Learning from Acoustic Features: audio2behavior](https://mlsanity.com/paper/1701.03198/) (2017)
- [MicroWalk: A Framework for Finding Side Channels in Binaries](https://mlsanity.com/paper/1808.05575/) (2018)
- [Encoding Power Traces as Images for Efficient Side-Channel Analysis](https://mlsanity.com/paper/2004.11015/) (2020)
- [Osiris: Automated Discovery of Microarchitectural Side Channels](https://mlsanity.com/paper/2106.03470/) (2021)

## Built on by (most cited first, 1 of 1)

- [Detokenization Leaks: Reconstructing Local LLM Outputs From Cache Traces](https://mlsanity.com/paper/2609.06674/) (2026)

## Abstract (arXiv)

The prosperous development of cloud computing and machine learning as a service has led to the widespread use of media software to process confidential media data. This paper explores an adversary's ability to launch side channel analyses (SCA) against media software to reconstruct confidential media inputs. Recent advances in representation learning and perceptual learning inspired us to consider the reconstruction of media inputs from side channel traces as a cross-modality manifold learning task that can be addressed in a unified manner with an autoencoder framework trained to learn the mapping between media inputs and side channel observations. We further enhance the autoencoder with attention to localize the program points that make the primary contribution to SCA, thus automatically pinpointing information-leakage points in media software. We also propose a novel and highly effective defensive technique called perception blinding that can perturb media inputs with perception masks and mitigate manifold learning-based SCA. Our evaluation exploits three popular media software to reconstruct inputs in image, audio, and text formats. We analyze three common side channels - cache bank, cache line, and page tables - and userspace-only cache set accesses logged by standard Prime+Probe. Our framework successfully reconstructs high-quality confidential inputs from the assessed media software and automatically pinpoint their vulnerable program points, many of which are unknown to the public. We further show that perception blinding can mitigate manifold learning-based SCA with negligible extra cost.