# S3: Spectral Null-Space Swap Makes Reasoning Models Efficient

arXiv: https://arxiv.org/abs/2609.37976 · HTML: https://mlsanity.com/paper/2609.37976/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Mixture-of-experts / [Mixture of LoRA experts](https://mlsanity.com/moe/mixture-of-lora-experts/)

## Builds on (most cited first, 20 of 20)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers](https://mlsanity.com/paper/2510.00915/) (2025)
- [Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens](https://mlsanity.com/paper/2602.13517/) (2026)
- [Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation](https://mlsanity.com/paper/2601.01984/) (2026)
- [EPO: Entropy-regularized Policy Optimization for LLM Agents Reinforcement Learning](https://mlsanity.com/paper/2509.22576/) (2025)
- [Heterogeneous Agent Collaborative Reinforcement Learning](https://mlsanity.com/paper/2603.02604/) (2026)
- [United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory](https://mlsanity.com/paper/2506.06843/) (2025)
- [VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction](https://mlsanity.com/paper/2602.12579/) (2026)
- [CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks](https://mlsanity.com/paper/2603.18736/) (2026)
- [RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics](https://mlsanity.com/paper/2603.12939/) (2026)
- [AEL: Evolving Agent Harness in Open-Ended Environments](https://mlsanity.com/paper/2604.21725/) (2026)
- [Uncertainty-Aware Reward Modeling for Stable RLHF](https://mlsanity.com/paper/2606.19818/) (2026)
- [StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback](https://mlsanity.com/paper/2508.06555/) (2025)
- [Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2606.02020/) (2026)
- [SocraticPO: Policy Optimization via Interactive Guidance](https://mlsanity.com/paper/2606.09887/) (2026)
- [Stop When Further Reasoning Won't Help: Attention-State Adaptive Generation in Reasoning Models](https://mlsanity.com/paper/2606.15070/) (2026)
- [Investigating and Alleviating Harm Amplification in LLM Interactions](https://mlsanity.com/paper/2606.02423/) (2026)
- [Spectral Rewiring for Exploration, Purification, and Model Merging](https://mlsanity.com/paper/2607.03065/) (2026)

## Abstract (arXiv)

LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model's weight component within the null space of a projection defined by the corresponding Non-thinking model's dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap (S3), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate S3 on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical, multimodal, and audio reasoning domains. S3 establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.