mlsanityData to 7 Oct 2026

Position: Capability Control Should be a Separate Goal From Alignment

arXiv 2602.05164 ↗First posted 5 Feb 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Foundation models are trained on broad data distributions, yielding generalist capabilities that enable many downstream applications but also expand the space of potential misuse and failures. This position paper argues that capability control -- imposing restrictions on permissible model behavior -- should be treated as a distinct goal from alignment. While alignment is often context and preference-driven, capability control aims to impose hard operational limits on permissible behaviors, including under adversarial elicitation. We organize capability control mechanisms across the model lifecycle into three layers: (i) data-based control of the training distribution, (ii) learning-based control via weight- or representation-level interventions, and (iii) system-based control via post-deployment guardrails over inputs, outputs, and actions. Because each layer has characteristic failure modes when used in isolation, we advocate for a defense-in-depth approach that composes complementary controls across the full stack. We further outline key open challenges in achieving such control, including the dual-use nature of knowledge and compositional generalization.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 110 of its 137 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

110 of 137 references.

References · 110

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2013Intriguing properties of neural networkscited by 5,114
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  13. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  14. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  15. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  16. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  17. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  18. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  19. 2019Machine Unlearningcited by 1,020
  20. 2016End to End Learning for Self-Driving Carscited by 1,000
  21. 2022Mass-Editing Memory in a Transformercited by 976
  22. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  23. 2023Steering Language Models With Activation Engineeringcited by 824
  24. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  25. 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
  26. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  27. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  28. 2024Scaling and evaluating sparse autoencoderscited by 530
  29. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  30. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  31. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  32. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  33. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  34. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  35. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  36. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  37. 2024Alignment faking in large language modelscited by 316
  38. 2018Reward learning from human preferences and demonstrations in Ataricited by 314

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.