mlsanityData to 7 Oct 2026

What Is the Alignment Tax?

arXiv 2603.00047 ↗First posted 9 Feb 2026econ.EM cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The alignment tax is widely discussed but has not been formally characterized. We provide a geometric theory of the alignment tax in representation space. Under linear representation assumptions, we define the alignment tax rate as the squared projection of the safety direction onto the capability subspace and derive the Pareto frontier governing safety-capability tradeoffs, parameterized by a single quantity of the principal angle between the safety and capability subspaces. We prove this frontier is tight and show it has a recursive structure. safety-safety tradeoffs under capability constraints are governed by the same equation, with the angle replaced by the partial correlation between safety objectives given capability directions. We derive a scaling law decomposing the alignment tax into an irreducible component determined by data structure and a packing residual that vanishes as O(m'/d) with model dimension d, and establish conditions under which capability preservation mediates or resolves conflicts between safety objectives.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 13 of its 26 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

13 of 26 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023Steering Language Models With Activation Engineeringcited by 824
  6. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  7. 2023Mitigating the Alignment Tax of RLHFcited by 181
  8. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  9. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  10. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  11. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  12. 2025Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Mergingcited by 10
1 more
  1. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models
  2. 2026Why Is RLHF Alignment Shallow? A Gradient Analysis
  3. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
  4. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
  5. 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
  6. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Control
  7. 2024Superficial Safety Alignment Hypothesis
  8. 2026BSO: Safety Alignment Is Density Ratio Matching