mlsanityData to 7 Oct 2026

Exploiting Novel GPT-4 APIs

arXiv 2312.14302 ↗First posted 21 Dec 2023cs.CR cs.AI cs.CLcited by 27

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Language model attacks typically assume one of two extreme threat models: full white-box access to model weights, or black-box access limited to a text generation API. However, real-world APIs are often more flexible than just text generation: these APIs expose "gray-box" access leading to new threat vectors. To explore this, we red-team three new functionalities exposed in the GPT-4 APIs: fine-tuning, function calling and knowledge retrieval. We find that fine-tuning a model on as few as 15 harmful examples or 100 benign examples can remove core safeguards from GPT-4, enabling a range of harmful outputs. Furthermore, we find that GPT-4 Assistants readily divulge the function call schema and can be made to execute arbitrary function calls. Finally, we find that knowledge retrieval can be hijacked by injecting instructions into retrieval documents. These vulnerabilities highlight that any additions to the functionality exposed by an API can create new vulnerabilities.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 26 of its 115 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

26 of 115 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
14 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  7. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  8. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  9. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  10. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  11. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 94
  12. 2023AI model GPT-3 (dis)informs us better than humanscited by 75
  13. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  14. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 8; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  6. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  8. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  9. 2024Safeguarding Large Language Models: A Surveycited by 73
  10. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  11. 2024Data-driven Discovery with Large Generative Modelscited by 33
  12. 2024Immunization against harmful fine-tuning attackscited by 33
15 more
  1. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  2. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  3. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  4. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  5. 2025AI Companies Should Report Pre- and Post-Mitigation Safety Evaluationscited by 3
  6. 2026Mitigating the OWASP Top 10 For Large Language Models Applications using Intelligent Agentscited by 3
  7. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  8. 2025Veracity: An Open-Source AI Fact-Checking Systemcited by 2
  9. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  10. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  11. 2025Securing generative artificial intelligence with parallel magnetic tunnel junction true randomnesscited by 1
  12. 2024"Moralized" Multi-Step Jailbreak Prompts: Black-Box Testing of Guardrails in Large Language Models for Verbal Attackscited by 0
  13. 2026Understanding the Effects of Safety Unalignment on Large Language Modelscited by 0
  14. 2026Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attackscited by 0
  15. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models
  2. 2026Control-Token Injection Suppresses Chain-of-Thought and Defeats Reasoning-Based Oversight in Tool-Using Agents
  3. 2026Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks
  4. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions
  5. 2025Security Concerns for Large Language Models: A Survey
  6. 2026DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation
  7. 2025An Empirical Study on the Security Vulnerabilities of GPTs ↗
  8. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Level