# Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle

arXiv: https://arxiv.org/abs/2407.13833 · HTML: https://mlsanity.com/paper/2407.13833/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 12 of 12)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?](https://mlsanity.com/paper/2404.14397/) (2024)
- [A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications](https://mlsanity.com/paper/2310.17750/) (2023)

## Built on by (most cited first, 30 of 32)

- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [GRIN: GRadient-INformed MoE](https://mlsanity.com/paper/2409.12136/) (2024)
- [The effect of fine-tuning on language model toxicity](https://mlsanity.com/paper/2410.15821/) (2024)
- [Beyond Linear Probes: Dynamic Safety Monitoring for Language Models](https://mlsanity.com/paper/2509.26238/) (2025)
- [Rethinking Safety in LLM Fine-tuning: An Optimization Perspective](https://mlsanity.com/paper/2508.12531/) (2025)
- [CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns](https://mlsanity.com/paper/2601.00588/) (2026)
- [When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Output](https://mlsanity.com/paper/2503.24191/) (2025)
- [AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreaks](https://mlsanity.com/paper/2504.12321/) (2025)
- [HearSay Benchmark: Do Audio LLMs Leak What They Hear?](https://mlsanity.com/paper/2601.03783/) (2026)
- [MedRiskEval: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settings](https://mlsanity.com/paper/2507.07248/) (2025)
- [Anecdoctoring: Automated Red-Teaming Across Language and Place](https://mlsanity.com/paper/2509.19143/) (2025)
- [Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges](https://mlsanity.com/paper/2511.01375/) (2025)
- [Learning to Extract Context for Context-Aware LLM Inference](https://mlsanity.com/paper/2512.11986/) (2025)
- [Large Language Models as Narrative-Driven Recommenders](https://mlsanity.com/paper/2410.13604/) (2024)
- [Mathematical Derivation Graphs: A Relation Extraction Task in STEM Manuscripts](https://mlsanity.com/paper/2410.21324/) (2024)
- [A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens](https://mlsanity.com/paper/2502.16366/) (2025)
- [Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours](https://mlsanity.com/paper/2605.04019/) (2026)
- [Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting](https://mlsanity.com/paper/2605.29498/) (2026)
- [OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Models](https://mlsanity.com/paper/2505.16036/) (2025)
- [Responsiveness Verification: Will Predictions Change? How Much? How Often?](https://mlsanity.com/paper/2507.02169/) (2025)
- [Moving Pictures of Thought: Extracting Visual Knowledge in Charles S. Peirce's Manuscripts with Vision-Language Models](https://mlsanity.com/paper/2511.13378/) (2025)
- [A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems](https://mlsanity.com/paper/2601.03992/) (2026)
- [Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross--Language Code Clone Detection](https://mlsanity.com/paper/2605.02860/) (2026)
- [Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping](https://mlsanity.com/paper/2605.04308/) (2026)
- [EinSort: Sorting is All We Need for Tensorizing LLM](https://mlsanity.com/paper/2606.08565/) (2026)
- [VISTA: Auditing Semantic Divergence in Vision-Language Models](https://mlsanity.com/paper/2607.02995/) (2026)
- [Securing LLMs in the Wild: Privacy and Security Challenges at the Edge](https://mlsanity.com/paper/2607.13088/) (2026)

## Abstract (arXiv)

Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3 series of language models. We utilized a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to cover a variety of harm areas in both single and multi-turn scenarios. Our results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks. Finally, we include additional red teaming strategies and evaluations that were used to test the safety behavior of Phi-3.5-mini and Phi-3.5-MoE, which were optimized for multilingual capabilities.