mlsanityData to 14 Oct 2026

Safety Pretraining: Toward the Next Generation of Safe AI

arXiv 2504.16980 ↗First posted 23 Apr 2025cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are brittle: once unsafe patterns are learned during pretraining, they are hard to remove. In this work, we present a data-centric pretraining framework that builds safety into the model from the start. Our framework consists of four key steps: (i) Safety Filtering: building a safety classifier to classify webdata into safe and unsafe categories; (ii) Safety Rephrasing: we recontextualize unsafe webdata into safer narratives; (iii) Native Refusal: we develop RefuseWeb and Moral Education pretraining datasets that actively teach model to refuse on unsafe content and the moral reasoning behind it, and (iv) Harmfulness-Tag annotated pretraining: we flag unsafe content during pretraining using a special token, and use it to steer model away from unsafe generations at inference. Our safety-pretrained models reduce attack success rates from 38.8% to 8.4% on standard LLM safety benchmarks with no performance degradation on general tasks.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

36 of 45 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
24 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,100
  2. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  3. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  10. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  12. 2023Pretraining Language Models with Human Preferencescited by 247
  13. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  14. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  15. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  16. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  17. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  18. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  19. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  20. 2024Arctic-Embed 2.0: Multilingual Retrieval Without Compromisecited by 58
  21. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  22. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  23. 2024Trojan Detection in Large Language Models: Insights from The Trojan Detection Challengecited by 14
  24. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 31.

43 papers.

Sort:

By area:

  1. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  2. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  3. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 43
  4. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  5. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 21
  6. 2026Model Spec Midtraining: Improving How Alignment Training Generalizescited by 20
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  8. 2026Shaping capabilities with token-level data filteringcited by 13
  9. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  10. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  11. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 9
  12. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
31 more
  1. 2026When Should We Introduce Safety Interventions During Pretraining?cited by 7
  2. 2026Tracing Persona Vectors Through LLM Pretrainingcited by 7
  3. 2025A Granular Study of Safety Pretraining under Model Abliterationcited by 6
  4. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  5. 2026Early Data Exposure Improves Robustness to Subsequent Fine-Tuningcited by 4
  6. 2026Constitutional Midtraining: Content Presence Drives Alignment Gainscited by 4
  7. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3
  8. 2026Reward Models Inherit Value Biases from Pretrainingcited by 3
  9. 2026Modular Pretraining Enables Access Controlcited by 3
  10. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 2
  11. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
  12. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  13. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alonecited by 2
  14. 2026Mark, Don't Erase: Token Inoculation for Dual-Use Knowledge in LLMscited by 2
  15. 2026Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balancecited by 1
  16. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stagescited by 1
  17. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflectioncited by 1
  18. 2026Synthetic Persona Pretraining: Alignment from Token Zerocited by 1
  19. 2025Towards provable probabilistic safety for scalable embodied AI systemscited by 0
  20. 2026Training a General Purpose Automated Red Teaming Modelcited by 0
  21. 2026Adversarial Reframing: A Framework for Targeted Generation in Language Modelscited by 0
  22. 2026Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safetycited by 0
  23. 2026Engineering Trustworthy Agentic AI for Critical Systemscited by 0
  24. 2026Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Modelscited by 0
  25. 2026Can Data Attribution Filter Out Subliminal Learning? Not Reliablycited by 0
  26. 2026A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponentscited by 0
  27. 2026Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervisioncited by 0
  28. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0
  29. 2026Pre-training interventions, ex post facto: Grafting model beliefs across checkpointscited by 0
  30. 2026Keep It CALM: Analyzing the Limits of Global Unsafety in Text-to-Image Generationcited by 0
  31. 2026When Protection Overrides Consequences: Understanding Protection Mode in Large Language Model Decision-Makingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing AI Safety with Source Code
  2. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models
  3. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
  4. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts
  5. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  6. 2026Trust The Typical
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
  8. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack