mlsanityData to 7 Oct 2026

Course-Correction: Safety Alignment Using Synthetic Preferences

arXiv 2407.16637 ↗First posted 23 Jul 2024cs.CL cs.AI cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The risk of harmful content generated by large language models (LLMs) becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of course-correction, ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the textscC2-Eval benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create textscC2-Syn, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, textscLlama2-Chat 7B and textscQwen2 7B, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 61 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

39 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  2. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States9 Jun
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
23 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  12. 2020Aligning AI With Shared Human Valuescited by 706
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  15. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  16. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  17. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  18. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  19. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  20. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  21. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  22. 2023Managing extreme AI risks amid rapid progresscited by 48
  23. 2024Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Biascited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  3. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  4. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  5. 2024Backtracking Improves Generation Safetycited by 31
  6. 2025Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findingscited by 10
  7. 2025FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domaincited by 8
  8. 2025AI Awarenesscited by 6
  9. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  10. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 2
  11. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  12. 2026Improving Safety Alignment via Balanced Direct Preference Optimizationcited by 1
1 more
  1. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  2. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  3. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
  4. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  5. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  6. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
  7. 2026An Efficient and Modular Framework for Targeted Harm Mitigation in LLMS
  8. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations