mlsanityData to 14 Oct 2026

The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

arXiv 2407.17915 ↗First posted 25 Jul 2024cs.CR cs.AIcited by 20

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities, but their power comes with significant security considerations. While extensive research has been conducted on the safety of LLMs in chat mode, the security implications of their function calling feature have been largely overlooked. This paper uncovers a critical vulnerability in the function calling process of LLMs, introducing a novel "jailbreak function" attack method that exploits alignment discrepancies, user coercion, and the absence of rigorous safety filters. Our empirical study, conducted on six state-of-the-art LLMs including GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-pro, reveals an alarming average success rate of over 90% for this attack. We provide a comprehensive analysis of why function calls are susceptible to such attacks and propose defensive strategies, including the use of defensive prompts. Our findings highlight the urgent need for enhanced security measures in the function calling capabilities of LLMs, contributing to the field of AI safety by identifying a previously unexplored risk, designing an effective attack method, and suggesting practical defensive measures. Our code is available at https://github.com/wooozihui/jailbreakfunction.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 30 of its 41 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

30 of 41 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2024Mixtral of Expertscited by 1,823
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  12. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
18 more
  1. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  2. 2020Aligning AI With Shared Human Valuescited by 723
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  6. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  10. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  11. 2023Pretraining Language Models with Human Preferencescited by 247
  12. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  13. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  14. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
  15. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 76
  16. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 41
  17. 2023Language Model Alignment with Elastic Resetcited by 34
  18. 2023Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethicscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  2. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  3. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
  4. 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 18
  5. 2026Clawdrain: Exploiting Tool-Calling Chains for Stealthy Token Exhaustion in OpenClaw Agentscited by 14
  6. 2025Measuring Harmfulness of Computer-Using Agentscited by 8
  7. 2025HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humorcited by 6
  8. 2024Generating a Low-code Complete Workflow via Task Decomposition and RAGcited by 3
  9. 2026Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agentscited by 3
  10. 2025StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Modelscited by 2
  11. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
  12. 2025AprielGuardcited by 2
8 more
  1. 2025Multi-Agent Framework for Threat Mitigation and Resilience in AI-Based Systemscited by 2
  2. 2026Involuntary In-Context Learning: Exploiting Few-Shot Pattern Completion to Bypass Safety Alignment in GPT-5.4cited by 2
  3. 2026Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Modelscited by 2
  4. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  5. 2025`A la recherche du sens perdu: your favourite LLM might have more to say than you can understandcited by 0
  6. 2026Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agentscited by 0
  7. 2026Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputscited by 0
  8. 2026Agent System Operations: Categorization, Challenges, and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Traces
  2. 2024Diversity Helps Jailbreak Large Language Models
  3. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  4. 2025Security Concerns for Large Language Models: A Survey
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  6. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
  7. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures
  8. 2026SoK: Robustness in Large Language Models against Jailbreak Attacks