mlsanityData to 7 Oct 2026

Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

arXiv 2501.09798 ↗First posted 16 Jan 2025cs.CR cs.CLcited by 15

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We surface a new threat to closed-weight Large Language Models (LLMs) that enables an attacker to compute optimization-based prompt injections. Specifically, we characterize how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor and allows developers to fine-tune LLMs for their tasks, thus providing utility, but also exposes enough information for an attacker to compute adversarial prompts. Through an experimental analysis, we characterize the loss-like values returned by the Gemini fine-tuning API and demonstrate that they provide a useful signal for discrete optimization of adversarial prompts using a greedy search algorithm. Using the PurpleLlama prompt injection benchmark, we demonstrate attack success rates between 65% and 82% on Google's Gemini family of LLMs. These attacks exploit the classic utility-security tradeoff - the fine-tuning interface provides a useful feature for developers but also exposes the LLMs to powerful attacks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 38 of its 64 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

38 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  2. 2024Fast Adversarial Attacks on Language Models In One GPU Minute23 Feb
  3. 2024Query-Based Adversarial Prompt Generation19 Feb
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  9. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 28

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  10. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  12. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
16 more
  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  2. 2023Poisoning Language Models During Instruction Tuningcited by 254
  3. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  4. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  5. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  6. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  7. 2024Stealing Part of a Production Language Modelcited by 152
  8. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  9. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  10. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 78
  11. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  12. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  13. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  14. 2024Instruction Tuning With Loss Over Instructionscited by 40
  15. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
  16. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Security Concerns for Large Language Models: A Surveycited by 26
  2. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
  3. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  4. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  5. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
  6. 2025RedTeamLLM: an Agentic AI framework for offensive securitycited by 6
  7. 2026An AI Agent Execution Environment to Safeguard User Datacited by 3
  8. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 3
  9. 2026Content Fuzzing for Escaping Information Cocoons on Digital Social Mediacited by 1
  10. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  11. 2026HijackKV: New Threat in Position-Independent KV Cache Reusecited by 1
  12. 2026Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systemscited by 1
3 more
  1. 2026A Sentence Relation-Based Approach to Sanitizing Malicious Instructionscited by 0
  2. 2026A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbationcited by 0
  3. 2026Arbitrary Cipher Attacks Against Large Language Models Do Not Require Fine-Tuningcited by 0