# Generating Fake Cyber Threat Intelligence Using Transformer-Based Models arXiv: https://arxiv.org/abs/2102.04351 ยท HTML: https://mlsanity.com/paper/2102.04351/ ## Key facts - Cited by 29 papers in the map (data through 14 Oct 2026). - Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. - Builds on 19 of its 58 references that are in the map. - First posted on arXiv 8 Feb 2021. ## Placed in - LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/) ## Builds on (most cited first, 19 of 19) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016) - [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019) - [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019) - [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019) - [Patent Claim Generation by Fine-Tuning OpenAI GPT-2](https://mlsanity.com/paper/1907.02052/) (2019) - [GenAug: Data Augmentation for Finetuning Text Generators](https://mlsanity.com/paper/2010.01794/) (2020) - [Two Stage Transformer Model for COVID-19 Fake News Detection and Fact Checking](https://mlsanity.com/paper/2011.13253/) (2020) - [RelExt: Relation Extraction using Deep Learning approaches for Cybersecurity Knowledge Graph Improvement](https://mlsanity.com/paper/1905.02497/) (2019) - [Thinking, Fast and Slow: Combining Vector Spaces and Knowledge Graphs](https://mlsanity.com/paper/1708.03310/) (2017) - [Cyber-All-Intel: An AI for Security related Threat Intelligence](https://mlsanity.com/paper/1905.02895/) (2019) - [A System for Automated Open-Source Threat Intelligence Gathering and Management](https://mlsanity.com/paper/2101.07769/) (2021) - [Mining Threat Intelligence about Open-Source Projects and Libraries from Code Repository Issues and Bug Reports](https://mlsanity.com/paper/1808.04673/) (2018) - [Preventing Poisoning Attacks on AI based Threat Intelligence Systems](https://mlsanity.com/paper/1807.07418/) (2018) - [Using Deep Neural Networks to Translate Multi-lingual Threat Intelligence](https://mlsanity.com/paper/1807.07517/) (2018) - [A System for Efficiently Hunting for Cyber Threats in Computer Systems Using Threat Intelligence](https://mlsanity.com/paper/2101.06761/) (2021) ## Built on by (most cited first, 29 of 29) - [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021) - [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023) - [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023) - [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022) - [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024) - [Provably Robust Multi-bit Watermarking for AI-generated Text](https://mlsanity.com/paper/2401.16820/) (2024) - [Beyond the Safeguards: Exploring the Security Risks of ChatGPT](https://mlsanity.com/paper/2305.08005/) (2023) - [Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity](https://mlsanity.com/paper/2401.07348/) (2024) - [G3Detector: General GPT-Generated Text Detector](https://mlsanity.com/paper/2305.12680/) (2023) - [AGIR: Automating Cyber Threat Intelligence Reporting with Natural Language Generation](https://mlsanity.com/paper/2310.02655/) (2023) - [A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models](https://mlsanity.com/paper/2311.11415/) (2023) - [Cyber Sentinel: Exploring Conversational Agents in Streamlining Security Tasks with GPT-4](https://mlsanity.com/paper/2309.16422/) (2023) - [Impacts and Risk of Generative AI Technology on Cyber Defense](https://mlsanity.com/paper/2306.13033/) (2023) - [On the Security Risks of Knowledge Graph Reasoning](https://mlsanity.com/paper/2305.02383/) (2023) - [From Texts to Shields: Convergence of Large Language Models and Cybersecurity](https://mlsanity.com/paper/2505.00841/) (2025) - [Controllable Fake Document Infilling for Cyber Deception](https://mlsanity.com/paper/2210.09917/) (2022) - [From Military to Healthcare: Adopting and Expanding Ethical Principles for Generative Artificial Intelligence](https://mlsanity.com/paper/2308.02448/) (2023) - [EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forums](https://mlsanity.com/paper/2411.05479/) (2024) - [False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systems](https://mlsanity.com/paper/2507.06252/) (2025) - [AI Security Threats against Pervasive Robotic Systems: A Course for Next Generation Cybersecurity Workforce](https://mlsanity.com/paper/2302.07953/) (2023) - [SecureReg: Combining NLP and MLP for Enhanced Detection of Malicious Domain Name Registrations](https://mlsanity.com/paper/2401.03196/) (2024) - [Contextualized AI for Cyber Defense: An Automated Survey using LLMs](https://mlsanity.com/paper/2409.13524/) (2024) - [Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation](https://mlsanity.com/paper/2607.05354/) (2026) - [Embracing the Generative AI Revolution: Advancing Tertiary Education in Cybersecurity with GPT](https://mlsanity.com/paper/2403.11402/) (2024) - [Who is Responsible When AI Fails? Mapping Causes, Entities, and Consequences of AI Privacy and Ethical Incidents](https://mlsanity.com/paper/2504.01029/) (2025) - [GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks](https://mlsanity.com/paper/2606.05844/) (2026) - [Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions](https://mlsanity.com/paper/2607.03233/) (2026) - [SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classification](https://mlsanity.com/paper/2607.09936/) (2026) - [Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA](https://mlsanity.com/paper/2607.18725/) (2026) ## Abstract (arXiv) Cyber-defense systems are being developed to automatically ingest Cyber Threat Intelligence (CTI) that contains semi-structured data and/or text to populate knowledge graphs. A potential risk is that fake CTI can be generated and spread through Open-Source Intelligence (OSINT) communities or on the Web to effect a data poisoning attack on these systems. Adversaries can use fake CTI examples as training input to subvert cyber defense systems, forcing the model to learn incorrect inputs to serve their malicious needs. In this paper, we automatically generate fake CTI text descriptions using transformers. We show that given an initial prompt sentence, a public language model like GPT-2 with fine-tuning, can generate plausible CTI text with the ability of corrupting cyber-defense systems. We utilize the generated fake CTI text to perform a data poisoning attack on a Cybersecurity Knowledge Graph (CKG) and a cybersecurity corpus. The poisoning attack introduced adverse impacts such as returning incorrect reasoning outputs, representation poisoning, and corruption of other dependent AI-based cyber defense systems. We evaluate with traditional approaches and conduct a human evaluation study with cybersecurity professionals and threat hunters. Based on the study, professional threat hunters were equally likely to consider our fake generated CTI as true.