mlsanityData to 7 Oct 2026

Modular Pretraining Enables Access Control

arXiv 2607.08077 ↗First posted 9 Jul 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

AI developers face a dual-use dilemma. An AI capability that helps one user cure a disease can help another synthesize one. This dilemma could be resolved with access control, limiting dual-use AI capabilities to trusted deployments with a legitimate need. A gold standard for access control would be to serve separate models with different capabilities to different users. However, training and deploying multiple models is prohibitively expensive. To address this challenge, we propose gradient-routed auxiliary modules (GRAM), a pre-training method that adds modules to a neural network and selectively updates them to induce specialization. Ablating a module at inference time removes its capability from the network, approximating a model trained on filtered data. We evaluate GRAM on synthetic stories and realistic dual-use data spanning virology, cybersecurity, nuclear physics, and specialized code. These experiments show that GRAM disables targeted capabilities while preserving the rest, and resists their recovery under finetuning better than post-hoc unlearning. Most importantly, a Chinchilla-optimal scaling analysis from 50M to 5B parameters shows that the gap between data-filtered and full-data models widens with scale on removed capabilities but stays small on retained ones, and that GRAM closely tracks data filtering. GRAM's training cost is independent of the number of supported capability profiles, yielding a 5x reduction over data filtering in our 5-profile setting.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

48 of 68 references.

References · 48

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  10. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  11. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
36 more
  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  2. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  3. 2024LoRA Learns Less and Forgets Lesscited by 313
  4. 2019Continual learning with hypernetworkscited by 267
  5. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  6. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  7. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  8. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  9. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  10. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  11. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  12. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  13. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  14. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  15. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  16. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  17. 2024The rising costs of training frontier AI modelscited by 91
  18. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  19. 2023Modular Deep Learningcited by 82
  20. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  21. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 54
  22. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  23. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
  24. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  25. 2023On Hate Scaling Laws For Data-Swampscited by 28
  26. 2024Scalable Language Model with Generalized Continual Learningcited by 28
  27. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  28. 2024Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Modelscited by 21
  29. 2025The Elicitation Game: Evaluating Capability Elicitation Techniquescited by 15
  30. 2025When Bad Data Leads to Good Modelscited by 11
  31. 2025Distillation Robustifies Unlearningcited by 11
  32. 2025Parameterized Synthetic Text Generation with SimpleStoriescited by 9
  33. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  34. 2025Access Controls Will Solve the Dual-Use Dilemmacited by 3
  35. 2025Studying Cross-cluster Modularity in Neural Networkscited by 1
  36. 2026Natively Unlearnable Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Latents to Wires: Surgical Post-Editing on Large Language Modelscited by 0