# Understanding and Exploiting Anisotropy in Post-Training

arXiv: https://arxiv.org/abs/2609.32792 · HTML: https://mlsanity.com/paper/2609.32792/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters](https://mlsanity.com/paper/2405.17604/) (2024)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference](https://mlsanity.com/paper/2408.08590/) (2024)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [Learning to Reason in 13 Parameters](https://mlsanity.com/paper/2602.04118/) (2026)
- [Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing](https://mlsanity.com/paper/2410.11462/) (2024)
- [Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning](https://mlsanity.com/paper/2410.19817/) (2024)
- [Rep2Text: Decoding Full Text from a Single LLM Token Representation](https://mlsanity.com/paper/2511.06571/) (2025)
- [Capsule Lens: Locating and Tracking Concept Geometry in Model Representations](https://mlsanity.com/paper/2609.05575/) (2026)
- [What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track](https://mlsanity.com/paper/2609.15064/) (2026)

## Abstract (arXiv)

LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective. Frequency-weighted likelihood training leaves a well-known signature: anisotropy, in which a few residual channels carry disproportionately large activations. Anisotropy is widely documented and usually treated as a defect, yet its function and its interaction with post-training remain unclear. We first analyze it. A label-free outlier rule isolates about 5% of residual channels that are essential for language modeling: removing them raises perplexity from 10 to over 106, versus 35 for count-matched random channels. Yet they barely distinguish correct from incorrect reasoning. SFT reshapes them, whereas RL leaves them largely intact and adapts the complementary channels. These channels therefore form the model's coherence substrate, and reasoning adaptation happens elsewhere. We then exploit this. textscSphereGate learns one bounded gain per residual channel on a frozen backbone. Its activation-weighted gradients provably limit movement of high-energy coherence channels and leave the remaining channels free. With 0.1M trainable parameters, textscSphereGate outperforms parameter-efficient baselines by 2.0--7.3 points on MATH-500 across Qwen2.5 (0.5B--7B) and Llama-3-8B, is comparable or exceeds full-model GRPO. Anisotropy is not a defect but a division of labor that post-training can exploit.