mlsanityData to 14 Oct 2026

TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification

arXiv 2402.12991 ↗First posted 20 Feb 2024cs.LG cs.AI cs.CLcited by 36

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Model (LLM) services and models often come with legal rules on who can use them and how they must use them. Assessing the compliance of the released LLMs is crucial, as these rules protect the interests of the LLM contributor and prevent misuse. In this context, we describe the novel fingerprinting problem of Black-box Identity Verification (BBIV). The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use. We repurpose adversarial suffixes, originally proposed for jailbreaking, to get a pre-defined answer from the target LLM, while other models give random answers. TRAP detects the target LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. TRAP remains effective even if the LLM has minor changes that do not significantly alter the original function.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 19 of its 41 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

19 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 18

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  7. 2018Hierarchical Neural Story Generationcited by 1,193
  8. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  9. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 496
  10. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  11. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  12. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 130
6 more
  1. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  2. 2023GPT-Sentinel: Distinguishing Human and ChatGPT Generated Contentcited by 43
  3. 2023Does GPT-4 pass the Turing test?cited by 37
  4. 2023Anatomy of an AI-powered malicious social botnetcited by 36
  5. 2023Towards Possibilities & Impossibilities of AI-generated Text Detection: A Surveycited by 28
  6. 2023Detecting ChatGPT: A Survey of the State of Detecting ChatGPT-Generated Textcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 22.

36 papers.

Sort:

By area:

  1. 2025Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trendscited by 20
  2. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 15
  3. 2025CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoorcited by 12
  4. 2025FIT-Print: Towards False-claim-resistant Model Ownership Verification via Targeted Fingerprintcited by 10
  5. 2025Fingerprinting LLMs via Prompt Injectioncited by 8
  6. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 8
  7. 2024MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Modelscited by 7
  8. 2025CoTSRF: Utilize Chain of Thought as Stealthy and Robust Fingerprint of Large Language Modelscited by 7
  9. 2026One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributionscited by 6
  10. 2025Peering Behind the Shield: Guardrail Identification in Large Language Modelscited by 5
  11. 2025The Challenge of Identifying the Origin of Black-Box Large Language Modelscited by 5
  12. 2026Token-Efficient Change Detection in LLM APIscited by 5
24 more
  1. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  2. 2025iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verificationcited by 4
  3. 2026AttnDiff: Attention-based Differential Fingerprinting for Large Language Modelscited by 4
  4. 2025AWM: Accurate Weight-Matrix Fingerprint for Large Language Modelscited by 3
  5. 2026KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditingcited by 3
  6. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  7. 2026Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Servicescited by 3
  8. 2025SELF: A Robust Singular Value and Eigenvalue Approach for LLM Fingerprintingcited by 2
  9. 2026Prompt2Fingerprint: Plug-and-Play LLM Fingerprinting via Text-to-Weight Generationcited by 2
  10. 2026Black-Box Forensics for Conversational LLM Agentscited by 2
  11. 2026Provable Model Provenance Set for Large Language Modelscited by 1
  12. 2026Implicit Identity Technologies for LLMs: Fingerprinting and Watermarking across Datasets, Models, and Generated Contentcited by 1
  13. 2026Fingerprinting Inference Systems of Large Language Modelscited by 1
  14. 2026Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verificationcited by 1
  15. 2026Black-Box Inference of LLM Architectural Properties with Restrictive API Accesscited by 0
  16. 2026Decision-Level Hijacking: Injecting Cognitive Bias into Large Language Models via Bit-Flip Attackscited by 0
  17. 2026Stemma: Induced Decision Regions Reveal LLM Provenancecited by 0
  18. 2026Who Built This Model? Tracing LLM Lineage via Spectral Fingerprints in Weight Spacecited by 0
  19. 2026AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probescited by 0
  20. 2026HoneyRoute: Honeypot-Model Routing for Adversarial LLM Servingcited by 0
  21. 2026Fingerprinting Multimodal Large Language Modelscited by 0
  22. 2026Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behaviorcited by 0
  23. 2026TP-CRIV: A Framework for Third-Party Challenge-Response Identity Verification of AI Modelscited by 0
  24. 2026Language Model Fingerprinting Requires Rethinking Watermark Teacherscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Black-box Optimization of LLM Outputs by Asking for Directions
  2. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information
  3. 2025Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
  4. 2023Certifying LLM Safety against Adversarial Prompting
  5. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs
  6. 2026An Empirical Study of Output-to-Input Loops for Black-Box Backdoor Detection in Fine-Tuned Open-Weight LLMs
  7. 2025Adversarial Suffix Filtering: a Defense Pipeline for LLMs
  8. 2026A Systematic Literature Review on LLM Defenses Against Prompt Injection and Jailbreaking: Expanding NIST Taxonomy