Weight-Space Geometry of Offline Reasoning Training
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone. We ask whether they are mechanistically distinct or converge to a similar weight update. Training six methods (SFT, RFT, DFT, RIFT, Offline GRPO, DPO) on identical math rollouts from a single base model (Qwen3-4B) with attention-only LoRA, we analyze the resulting deltas via cosine similarity, principal-angle subspace analysis, linear mode connectivity, and CKA. We observe: (i) SFT, RFT, and RIFT have nearly colinear weight deltas (cosine >= 0.97, top-1 principal angle ~7 deg median over 144 modules) and comparable GSM8K accuracy (87-88%, n=1319; pairwise McNemar p >= 0.15); (ii) DFT diverges further in direction than any reward-weighted method despite using the same data; (iii) Offline GRPO adds a substantial component orthogonal to the SFT direction (~67% globally, up to ~86% in late layers) while staying in the SFT loss basin; (iv) DPO sits in a near-orthogonal subspace, shows a mode-connectivity barrier, and collapses late-layer CKA to ~0.46. DPO also reaches the highest accuracy in our protocol on both GSM8K (93.5%, McNemar p < 10^-9 vs. each other method) and AIME26 (30.0% vs. 3.3-10.0%); its training uses a 10x smaller learning rate than the others (the standard convention), so the update-norm and accuracy gaps reflect loss-function and optimizer choices jointly, and a learning-rate-matched DPO comparison is left for future work.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 18 references that are in the map. First posted on arXiv 21 Jun 2026.
Builds on
13 of 18 references.
Closest ancestors · 3
Same direction, newest first
- 2025Learning to Reason under Off-Policy Guidance21 Apr
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
Other references · 10
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
- 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
- 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
- 2025Rank-1 LoRAs Encode Interpretable Reasoning Signalscited by 4
- 2026RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoning
- 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
- 2025Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning
- 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
- 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
- 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
- 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
- 2024Simple Ingredients for Offline Reinforcement Learning