mlsanityData to 14 Oct 2026

Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message

arXiv 2507.04673 ↗First posted 7 Jul 2025cs.AIcited by 2

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The rise of conversational interfaces has greatly enhanced LLM usability by leveraging dialogue history for sophisticated reasoning. However, this reliance introduces an unexplored attack surface. This paper introduces Trojan Horse Prompting, a novel jailbreak technique. Adversaries bypass safety mechanisms by forging the model's own past utterances within the conversational history provided to its API. A malicious payload is injected into a model-attributed message, followed by a benign user prompt to trigger harmful content generation. This vulnerability stems from Asymmetric Safety Alignment: models are extensively trained to refuse harmful user requests but lack comparable skepticism towards their own purported conversational history. This implicit trust in its "past" creates a high-impact vulnerability. Experimental validation on Google's Gemini-2.0-flash-preview-image-generation shows Trojan Horse Prompting achieves a significantly higher Attack Success Rate (ASR) than established user-turn jailbreaking methods. These findings reveal a fundamental flaw in modern conversational AI security, necessitating a paradigm shift from input-level filtering to robust, protocol-level validation of conversational context integrity.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 21 of its 30 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

21 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 18

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  8. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  9. 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 189
  10. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 57
  11. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistencycited by 51
  12. 2024HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Modelscited by 16
6 more
  1. 2025Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoningcited by 15
  2. 2024Gradient-based Jailbreak Images for Multimodal Fusion Modelscited by 14
  3. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  4. 2024Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Stepcited by 11
  5. 2025GenBreak: Red Teaming Text-to-Image Generators Using Large Language Modelscited by 4
  6. 2024Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamicscited by 1
  2. 2026Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation
  2. 2025Jailbreaking is (Mostly) Simpler Than You Think
  3. 2025Automating Deception: Scalable Multi-Turn LLM Jailbreaks
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs
  5. 2026Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue
  6. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Models
  7. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
  8. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models