mlsanityData to 7 Oct 2026

Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle

arXiv 2407.13833 ↗First posted 18 Jul 2024cs.CL cs.AIcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3 series of language models. We utilized a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to cover a variety of harm areas in both single and multi-turn scenarios. Our results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks. Finally, we include additional red teaming strategies and evaluations that were used to test the safety behavior of Phi-3.5-mini and Phi-3.5-MoE, which were optimized for multilingual capabilities.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 19 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

12 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 11

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Textbooks Are All You Needcited by 561
  4. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 34
  11. 2023A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applicationscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 12.

32 papers.

Sort:

By area:

  1. 2024Phi-4 Technical Reportcited by 675
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  3. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  4. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  5. 2024GRIN: GRadient-INformed MoEcited by 16
  6. 2024The effect of fine-tuning on language model toxicitycited by 15
  7. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 14
  8. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
  9. 2026CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patternscited by 5
  10. 2025When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Outputcited by 3
  11. 2025AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreakscited by 3
  12. 2025MedRiskEval: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settingscited by 2
20 more
  1. 2025Anecdoctoring: Automated Red-Teaming Across Language and Placecited by 2
  2. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judgescited by 2
  3. 2024Large Language Models as Narrative-Driven Recommenderscited by 1
  4. 2024Mathematical Derivation Graphs: A Relation Extraction Task in STEM Manuscriptscited by 1
  5. 2025A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokenscited by 1
  6. 2025Learning to Extract Context for Context-Aware LLM Inferencecited by 1
  7. 2026HearSay Benchmark: Do Audio LLMs Leak What They Hear?cited by 1
  8. 2026Redefining AI Red Teaming in the Agentic Era: From Weeks to Hourscited by 1
  9. 2026Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgettingcited by 1
  10. 2025OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Modelscited by 0
  11. 2025Responsiveness Verification: Will Predictions Change? How Much? How Often?cited by 0
  12. 2025Moving Pictures of Thought: Extracting Visual Knowledge in Charles S. Peirce's Manuscripts with Vision-Language Modelscited by 0
  13. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systemscited by 0
  14. 2026Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detectioncited by 0
  15. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  16. 2026EinSort: Sorting is All We Need for Tensorizing LLMcited by 0
  17. 2026VISTA: Auditing Semantic Divergence in Vision-Language Modelscited by 0
  18. 2026Securing LLMs in the Wild: Privacy and Security Challenges at the Edgecited by 0
  19. 2026SRD-GUARD: A Defense Framework of LLMs via Semantic Rewriting and Joint Multi-Model Scoring for Latent Intent Exposurecited by 0
  20. 2026TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
  2. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
  4. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
  5. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  6. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  7. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements
  8. 2026BabelSteering: Multilingual Safety Alignment via English Steering Vectors