# StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation

arXiv: https://arxiv.org/abs/2601.04740 · HTML: https://mlsanity.com/paper/2601.04740/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic](https://mlsanity.com/paper/2402.11746/) (2024)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing](https://mlsanity.com/paper/2402.16192/) (2024)
- [Jointly Measuring Diversity and Quality in Text Generation Models](https://mlsanity.com/paper/1904.03971/) (2019)
- [Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs](https://mlsanity.com/paper/2402.14872/) (2024)
- [h4rm3l: A language for Composable Jailbreak Attack Synthesis](https://mlsanity.com/paper/2408.04811/) (2024)
- [RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems](https://mlsanity.com/paper/2409.16727/) (2024)
- [Prompt Injection 2.0: Hybrid AI Threats](https://mlsanity.com/paper/2507.13169/) (2025)
- [Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction](https://mlsanity.com/paper/2502.11084/) (2025)
- [Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?](https://mlsanity.com/paper/2509.06350/) (2025)
- [Understanding and Mitigating Risks of Generative AI in Financial Services](https://mlsanity.com/paper/2504.20086/) (2025)
- [MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning](https://mlsanity.com/paper/2506.16792/) (2025)
- [The Resurgence of GCG Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2509.00391/) (2025)

## Built on by (most cited first, 4 of 4)

- [KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?](https://mlsanity.com/paper/2601.13240/) (2026)
- [From Cheap Fakes to Pure Synthesis: Addressing the New Era of T2V Fake News Videos](https://mlsanity.com/paper/2608.06732/) (2026)
- [EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models](https://mlsanity.com/paper/2609.26175/) (2026)
- [FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance](https://mlsanity.com/paper/2609.36474/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly applied in specialized domains such as finance and healthcare, where they introduce unique safety risks. Domain-specific datasets of harmful prompts remain scarce and still largely rely on manual construction; public datasets mainly focus on explicit harmful prompts, which modern LLM defenses can often detect and refuse. In contrast, implicit harmful prompts-expressed through indirect domain knowledge-are harder to detect and better reflect real-world threats. We identify two challenges: transforming domain knowledge into actionable constraints and increasing the implicitness of generated harmful prompts. To address them, we propose an end-to-end framework that first performs knowledge-graph-guided harmful prompt generation to systematically produce domain-relevant prompts, and then applies two-strategy obfuscation rewriting to convert explicit harmful prompts into implicit variants via direct and context-enhanced rewriting. This framework yields high-quality datasets combining strong domain relevance with implicitness, enabling more realistic red-teaming and advancing LLM safety research. We release our code and datasets on GitHub.