# Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining

arXiv: https://arxiv.org/abs/2410.18371 · HTML: https://mlsanity.com/paper/2410.18371/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [One-Class Classification: Taxonomy of Study and Review of Techniques](https://mlsanity.com/paper/1312.0049/) (2013)
- [Hear "No Evil", See "Kenansville": Efficient and Transferable Black-Box Attacks on Speech Recognition and Voice Identification Systems](https://mlsanity.com/paper/1910.05262/) (2019)
- [Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study](https://mlsanity.com/paper/2310.00108/) (2023)
- [M^4I: Multi-modal Models Membership Inference](https://mlsanity.com/paper/2209.06997/) (2022)
- [Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning](https://mlsanity.com/paper/2407.16307/) (2024)
- [Does CLIP Know My Face?](https://mlsanity.com/paper/2209.07341/) (2022)
- [Membership Inference Attacks Against Self-supervised Speech Models](https://mlsanity.com/paper/2111.05113/) (2021)
- [SLMIA-SR: Speaker-Level Membership Inference Attacks against Speaker Recognition Systems](https://mlsanity.com/paper/2309.07983/) (2023)
- [Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training](https://mlsanity.com/paper/2408.07919/) (2024)
- [User-Level Differential Privacy against Attribute Inference Attack of Speech Emotion Recognition in Federated Learning](https://mlsanity.com/paper/2204.02500/) (2022)

## Built on by (most cited first, 10 of 10)

- [Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models](https://mlsanity.com/paper/2412.05934/) (2024)
- [Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment](https://mlsanity.com/paper/2503.18991/) (2025)
- [PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization](https://mlsanity.com/paper/2412.05892/) (2024)
- [A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming](https://mlsanity.com/paper/2505.24252/) (2025)
- [Assessing the Effectiveness of Membership Inference on Generative Music](https://mlsanity.com/paper/2512.21762/) (2025)
- [DMRL: Data- and Model-aware Reward Learning for Data Extraction](https://mlsanity.com/paper/2505.06284/) (2025)
- [VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Search](https://mlsanity.com/paper/2510.15948/) (2025)
- [Information-Dense Reasoning for Efficient and Auditable Security Alert Triage](https://mlsanity.com/paper/2512.08169/) (2025)
- [ShellForge: Adversarial Co-Evolution of Webshell Generation and Multi-View Detection for Robust Webshell Defense](https://mlsanity.com/paper/2601.22182/) (2026)
- [From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop](https://mlsanity.com/paper/2608.11171/) (2026)

## Abstract (arXiv)

Audio can disclose PII, particularly when combined with related text data. Therefore, it is essential to develop tools to detect privacy leakage in Contrastive Language-Audio Pretraining(CLAP). Existing MIAs need audio as input, risking exposure of voiceprint and requiring costly shadow models. We first propose PRMID, a membership inference detector based probability ranking given by CLAP, which does not require training shadow models but still requires both audio and text of the individual as input. To address these limitations, we then propose USMID, a textual unimodal speaker-level membership inference detector, querying the target model using only text data. We randomly generate textual gibberish that are clearly not in training dataset. Then we extract feature vectors from these texts using the CLAP model and train a set of anomaly detectors on them. During inference, the feature vector of each test text is input into the anomaly detector to determine if the speaker is in the training set (anomalous) or not (normal). If available, USMID can further enhance detection by integrating real audio of the tested speaker. Extensive experiments on various CLAP model architectures and datasets demonstrate that USMID outperforms baseline methods using only text data.