mlsanityData to 7 Oct 2026

Multi-step Jailbreaking Privacy Attacks on ChatGPT

arXiv 2304.05197 ↗First posted 11 Apr 2023cs.CL cs.CRcited by 320

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

With the rapid progress of large language models (LLMs), many downstream NLP tasks can be well solved given appropriate prompts. Though model developers and researchers work hard on dialog safety to avoid generating harmful content from LLMs, it is still challenging to steer AI-generated content (AIGC) for the human good. As powerful LLMs are devouring existing text data from various domains (e.g., GPT-3 is trained on 45TB texts), it is natural to doubt whether the private information is included in the training data and what privacy threats can these LLMs and their downstream applications bring. In this paper, we study the privacy threats from OpenAI's ChatGPT and the New Bing enhanced by ChatGPT and show that application-integrated LLMs may cause new privacy threats. To this end, we conduct extensive experiments to support our claims and discuss LLMs' privacy implications.

Key facts. Cited by 320 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 25 of its 44 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

25 of 44 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
13 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  6. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  8. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  9. 2020Information Leakage in Embedding Modelscited by 199
  10. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  11. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  12. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  13. 2022You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personascited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 78; 2024: 129; 2025: 86; 2026: 27.

320 papers.

Sort:

By area:

  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  12. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
38 more
  1. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  6. 2023Large Language Model Alignment: A Surveycited by 241
  7. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  9. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  10. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  11. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  12. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  13. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  14. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  15. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  16. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  17. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  18. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  19. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  20. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  21. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  22. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  23. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  24. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  25. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  26. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  27. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  28. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  29. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  30. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 97
  31. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  32. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  33. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  34. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  35. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  36. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  37. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  38. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81

50 most cited of 320 All 320 (JSON)