Position: Capability Control Should be a Separate Goal From Alignment
Not on the map. Why?
Abstract (arXiv)
Foundation models are trained on broad data distributions, yielding generalist capabilities that enable many downstream applications but also expand the space of potential misuse and failures. This position paper argues that capability control -- imposing restrictions on permissible model behavior -- should be treated as a distinct goal from alignment. While alignment is often context and preference-driven, capability control aims to impose hard operational limits on permissible behaviors, including under adversarial elicitation. We organize capability control mechanisms across the model lifecycle into three layers: (i) data-based control of the training distribution, (ii) learning-based control via weight- or representation-level interventions, and (iii) system-based control via post-deployment guardrails over inputs, outputs, and actions. Because each layer has characteristic failure modes when used in isolation, we advocate for a defense-in-depth approach that composes complementary controls across the full stack. We further outline key open challenges in achieving such control, including the dual-use nature of knowledge and compositional generalization.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 110 of its 137 references that are in the map. First posted on arXiv 5 Feb 2026.
Builds on
110 of 137 references.
References · 110
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2013Intriguing properties of neural networkscited by 5,114
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2025s1: Simple test-time scalingcited by 1,311
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2019Machine Unlearningcited by 1,020
- 2016End to End Learning for Self-Driving Carscited by 1,000
- 2022Mass-Editing Memory in a Transformercited by 976
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
- 2022FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operatorscited by 748
- 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
- 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
- 2025Reasoning Models Don't Always Say What They Thinkcited by 388
- 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
- 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
- 2024Alignment faking in large language modelscited by 316
- 2018Reward learning from human preferences and demonstrations in Ataricited by 314
50 most cited of 110 All 110 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.