mlsanityData to 7 Oct 2026

Atoxia: Red-teaming Large Language Models with Target Toxic Answers

arXiv 2408.14853 ↗First posted 27 Aug 2024cs.CL cs.AI cs.CRcited by 7

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Despite the substantial advancements in artificial intelligence, large language models (LLMs) remain being challenged by generation safety. With adversarial jailbreaking prompts, one can effortlessly induce LLMs to output harmful content, causing unexpected negative social impacts. This vulnerability highlights the necessity for robust LLM red-teaming strategies to identify and mitigate such risks before large-scale application. To detect specific types of risks, we propose a novel red-teaming method that Attacks LLMs with Target Toxic Answers (Atoxia). Given a particular harmful answer, Atoxia generates a corresponding user query and a misleading answer opening to examine the internal defects of a given LLM. The proposed attacker is trained within a reinforcement learning scheme with the LLM outputting probability of the target answer as the reward. We verify the effectiveness of our method on various red-teaming benchmarks, such as AdvBench and HH-Harmless. The empirical results demonstrate that Atoxia can successfully detect safety risks in not only open-source models but also state-of-the-art black-box models such as GPT-4o.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 31 of its 44 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

31 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity30 Jan
  2. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
17 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  9. 2023Using an LLM to Help With Code Understandingcited by 222
  10. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  11. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  12. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  13. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  14. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  15. 2022Graph Enhanced Contrastive Learning for Radiology Findings Summarizationcited by 24
  16. 2023Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformerscited by 9
  17. 2024LLMs Could Autonomously Learn Without External Supervisioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignmentcited by 14
  2. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMscited by 5
  3. 2025Toward a Dynamic Stackelberg Game-Theoretic Framework for Agentic AI Defense Against LLM Jailbreakingcited by 5
  4. 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 3
  5. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  6. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  7. 2026VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inferencecited by 0