mlsanityData to 7 Oct 2026

On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?

arXiv 2310.01581 ↗First posted 2 Oct 2023cs.LG cs.AI cs.CRcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) have achieved unprecedented performance in Natural Language Generation (NLG) tasks. However, many existing studies have shown that they could be misused to generate undesired content. In response, before releasing LLMs for public access, model developers usually align those language models through Supervised Fine-Tuning (SFT) or Reinforcement Learning with Human Feedback (RLHF). Consequently, those aligned large language models refuse to generate undesired content when facing potentially harmful/unethical requests. A natural question is "could alignment really prevent those open-sourced large language models from being misused to generate undesired content?''. In this work, we provide a negative answer to this question. In particular, we show those open-sourced, aligned large language models could be easily misguided to generate undesired content without heavy computations or careful prompt designs. Our key idea is to directly manipulate the generation process of open-sourced LLMs to misguide it to generate undesired content including harmful or biased information and even private data. We evaluate our method on 4 open-sourced LLMs accessible publicly and our finding highlights the need for more advanced mitigation strategies for open-sourced LLMs.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 38 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

29 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
12 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  4. 2021Editing Factual Knowledge in Language Modelscited by 548
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  6. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  8. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  9. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  10. 2021Adaptive Machine Unlearningcited by 164
  11. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  12. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 14; 2025: 7; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  3. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 65
  6. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attackscited by 48
  7. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  8. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  9. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  10. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  11. 2024Extracting Unlearned Information from LLMs with Activation Steeringcited by 19
  12. 2024Permute-and-Flip: An optimally stable and watermarkable decoder for LLMscited by 17
13 more
  1. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  2. 2024Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxiacited by 12
  3. 2024On the (In)Security of LLM App Storescited by 12
  4. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  5. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  6. 2025You Can't Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectorscited by 4
  7. 2024Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attackscited by 2
  8. 2025Many-Turn Jailbreakingcited by 2
  9. 2025Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integrationcited by 1
  10. 2025TestDG: Test-time Domain Generalization for Continual Test-time Adaptationcited by 0
  11. 2025AJF: Adaptive Jailbreak Framework Based on the Comprehension Ability of Black-Box Large Language Modelscited by 0
  12. 2026Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputscited by 0
  13. 2026BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  2. 2023Large Language Model Unlearning
  3. 2025A Survey on Training-free Alignment of Large Language Models
  4. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2023Aligning Large Language Models with Human: A Survey
  8. 2024ABC Align: Large Language Model Alignment for Safety & Accuracy