mlsanityData to 14 Oct 2026

GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis

arXiv 2510.09260 ↗First posted 10 Oct 2025cs.CR cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Recent work has shown that RLHF is highly susceptible to backdoor attacks. However, existing methods often rely on rare tokens or fixed triggers, limiting their impact in realistic scenarios. In this work, we develop GREAT, a novel framework for crafting natural distributional backdoors in RLHF. Specifically, GREAT targets harmful response generation for a vulnerable user subpopulation featured by semantically violent requests paired with emotionally angry triggers. At the core of our framework is a trigger identification pipeline that operates in the model's latent embedding space, leveraging dimensionality reduction and clustering techniques to identify representative triggers. To enable this, we introduce a hierarchical and diversity-driven prompting strategy to construct Erinyes, a high-quality dataset of over 5,000 angry triggers curated from GPT-4.1. Our experiments show that GREAT significantly outperforms baselines in attack generalization to unseen triggers, while preserving standard utility and maintaining stealth under defenses.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 87 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

45 of 87 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data8 Apr
  2. 2024Preference Poisoning Attacks on Reward Model Learning2 Feb
  3. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  4. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models16 Nov

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2017A Closer Look at Memorization in Deep Networkscited by 942
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
29 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  2. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  3. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  4. 2018Spectral Signatures in Backdoor Attackscited by 484
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  6. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 456
  7. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  8. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  9. 2023Poisoning Language Models During Instruction Tuningcited by 269
  10. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  11. 2019A backdoor attack against LSTM-based text classification systemscited by 227
  12. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  13. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  14. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
  15. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 94
  16. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 80
  17. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  18. 2020Subpopulation Data Poisoning Attackscited by 70
  19. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 66
  20. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  21. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 57
  22. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 53
  23. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  24. 2024Here's a Free Lunch: Sanitizing Backdoored Models with Model Mergecited by 26
  25. 2021Excess Capacity and Backdoor Poisoningcited by 24
  26. 2024TuBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuningcited by 14
  27. 2023Backdoor Attacks with Input-unique Triggers in NLPcited by 12
  28. 2024Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attackscited by 1
  29. 2024Generalizable Targeted Data Poisoning against Varying Physical Objectscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026OPBackdoor: Opportunistic Backdoors via Alibi-Aligned Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs
  2. 2026Lilith: Backdoor Generalization under Training-Inference Trigger Shift
  3. 2026When Emotion Becomes Trigger: Emotion-style dynamic Backdoor Attack Parasitising Large Language Models
  4. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs
  5. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMs
  6. 2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models
  7. 2026Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggers
  8. 2026Backdoor Unlearning Generalization: A Path Toward the Removal of Unknown Triggers in LLMs