mlsanityData to 7 Oct 2026

When Should We Introduce Safety Interventions During Pretraining?

arXiv 2601.07087 ↗First posted 11 Jan 2026cs.LGcited by 5

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Prior work has shown that safety interventions applied during pretraining, such as removing and rephrasing harmful content, can substantially improve the robustness of the resulting models. In this paper, we study the fundamental question that prior work has overlooked: "When during pretraining should safety interventions be introduced?" We keep the underlying data sources and pretraining interventions fixed, varying the intervention start time (after 0%, 20%, or 60% of pretraining tokens). We find that the optimal start time is not one-size-fits-all: with standard top-k decoding, introducing interventions after a short initial phase of safe-only pretraining (20%-60%) often yields the strongest robustness, with the clearest benefits emerging after downstream, benign finetuning. In contrast, for safety-aware inference, interventions starting from the beginning improve steerability towards safer generations. Finally, we observe that earlier interventions reshape internal representations: linear probes more cleanly separate safe vs harmful examples. Our results are the first to establish intervention timing as a key curriculum design choice for safety.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 40 of its 51 references that are in the map. First posted on arXiv 11 Jan 2026.

Builds on

40 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  2. 2024Improving Alignment and Robustness with Circuit Breakers6 Jun
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning5 Mar

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  12. 2023Steering Language Models With Activation Engineeringcited by 824
25 more
  1. 2023Textbooks Are All You Needcited by 561
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  5. 2017Automated Curriculum Learning for Neural Networkscited by 251
  6. 2023Pretraining Language Models with Human Preferencescited by 245
  7. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  8. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 212
  9. 2022Can large language models reason about medical questions?cited by 194
  10. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  11. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  12. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  13. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  14. 2020When Do Curricula Work?cited by 105
  15. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  16. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  17. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  18. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  19. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  20. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
  21. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  22. 2024Trojan Detection in Large Language Models: Insights from The Trojan Detection Challengecited by 14
  23. 2025When Bad Data Leads to Good Modelscited by 11
  24. 2025Analyzing Similarity Metrics for Data Selection for Language Model Pretrainingcited by 3
  25. 2025Evaluating Language Model Reasoning about Confidential Informationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  2. 2026The Finetuner's Fallacy: When to Pretrain with Your Finetuning Datacited by 7
  3. 2026Early Data Exposure Improves Robustness to Subsequent Fine-Tuningcited by 3
  4. 2026Constitutional Midtraining: Content Presence Drives Alignment Gainscited by 3
  5. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflectioncited by 0