# TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification

arXiv: https://arxiv.org/abs/2402.12991 · HTML: https://mlsanity.com/paper/2402.12991/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Jailbreaking, one of 12 directions in LLM safety and security.
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Placed in

- LLM safety and security / [Jailbreaking](https://mlsanity.com/safety/jailbreaking/)

## Builds on (most cited first, 19 of 19)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding](https://mlsanity.com/paper/2009.03015/) (2020)
- [On the Possibilities of AI-Generated Text Detection](https://mlsanity.com/paper/2304.04736/) (2023)
- [GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content](https://mlsanity.com/paper/2305.07969/) (2023)
- [Does GPT-4 pass the Turing test?](https://mlsanity.com/paper/2310.20216/) (2023)
- [Anatomy of an AI-powered malicious social botnet](https://mlsanity.com/paper/2307.16336/) (2023)
- [Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey](https://mlsanity.com/paper/2310.15264/) (2023)
- [Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Text](https://mlsanity.com/paper/2309.07689/) (2023)

## Built on by (most cited first, 30 of 36)

- [Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends](https://mlsanity.com/paper/2508.11548/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor](https://mlsanity.com/paper/2509.09703/) (2025)
- [FIT-Print: Towards False-claim-resistant Model Ownership Verification via Targeted Fingerprint](https://mlsanity.com/paper/2501.15509/) (2025)
- [Fingerprinting LLMs via Prompt Injection](https://mlsanity.com/paper/2509.25448/) (2025)
- [Are Robust LLM Fingerprints Adversarially Robust?](https://mlsanity.com/paper/2509.26598/) (2025)
- [MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Models](https://mlsanity.com/paper/2410.08604/) (2024)
- [CoTSRF: Utilize Chain of Thought as Stealthy and Robust Fingerprint of Large Language Models](https://mlsanity.com/paper/2505.16785/) (2025)
- [One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions](https://mlsanity.com/paper/2607.10252/) (2026)
- [Peering Behind the Shield: Guardrail Identification in Large Language Models](https://mlsanity.com/paper/2502.01241/) (2025)
- [The Challenge of Identifying the Origin of Black-Box Large Language Models](https://mlsanity.com/paper/2503.04332/) (2025)
- [Token-Efficient Change Detection in LLM APIs](https://mlsanity.com/paper/2602.11083/) (2026)
- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verification](https://mlsanity.com/paper/2511.08905/) (2025)
- [AttnDiff: Attention-based Differential Fingerprinting for Large Language Models](https://mlsanity.com/paper/2604.05502/) (2026)
- [AWM: Accurate Weight-Matrix Fingerprint for Large Language Models](https://mlsanity.com/paper/2510.06738/) (2025)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences](https://mlsanity.com/paper/2606.03330/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprinting](https://mlsanity.com/paper/2512.03620/) (2025)
- [Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generation](https://mlsanity.com/paper/2605.18474/) (2026)
- [Black-Box Forensics for Conversational LLM Agents](https://mlsanity.com/paper/2606.22698/) (2026)
- [Provable Model Provenance Set for Large Language Models](https://mlsanity.com/paper/2602.00772/) (2026)
- [Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Content](https://mlsanity.com/paper/2605.29245/) (2026)
- [Fingerprinting Inference Systems of Large Language Models](https://mlsanity.com/paper/2605.29979/) (2026)
- [Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verification](https://mlsanity.com/paper/2608.08195/) (2026)
- [Black-Box Inference of LLM Architectural Properties with Restrictive API Access](https://mlsanity.com/paper/2607.01313/) (2026)
- [Decision-Level Hijacking: Injecting Cognitive Bias into Large Language Models via Bit-Flip Attacks](https://mlsanity.com/paper/2607.25227/) (2026)
- [Stemma: Induced Decision Regions Reveal LLM Provenance](https://mlsanity.com/paper/2607.25880/) (2026)
- [Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Space](https://mlsanity.com/paper/2608.07786/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. Assessing the compliance of the released LLMs is crucial, as these rules protect the interests of the LLM contributor and prevent misuse. In this context, we describe the novel fingerprinting problem of Black-box Identity Verification (BBIV). The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use. We repurpose adversarial suffixes, originally proposed for jailbreaking, to get a pre-defined answer from the target LLM, while other models give random answers. TRAP detects the target LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. TRAP remains effective even if the LLM has minor changes that do not significantly alter the original function.