mlsanityData to 14 Oct 2026

Reasoning-Finetuning Repurposes Latent Representations in Base Models

arXiv 2507.12638 ↗First posted 16 Jul 2025cs.LGcited by 26

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Backtracking, an emergent behavior elicited by reasoning fine-tuning, has been shown to be a key mechanism in reasoning models' enhanced capabilities. Prior work has succeeded in manipulating this behavior via steering vectors, but the underlying mechanism remains poorly understood. In this work, we show that the emergence of backtracking in DeepSeek-R1-Distill-Llama-8B is in part driven by a repurposed direction already present in base model activations. Specifically, we identify a direction in base Llama-3.1-8B's residual stream which systematically induces backtracking when used to steer the distilled reasoning model, and find that the effects of steering with this direction cannot be trivially explained by token-level attributes. We further find that this direction does not induce backtracking in the base model, suggesting that the reasoning finetuning process repurposes pre-existing representations to form new behavioral circuits. Additionally, we hypothesize that this direction is one of several which may work together to mediate backtracking. Our findings offer a compelling picture that reasoning-finetuned models repurpose pre-existing base model representations, rather than learn new capabilities from scratch.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 8 of its 12 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

8 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Understanding Reasoning in Thinking Language Models via Steering Vectors22 Jun
  2. 2025Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Models6 Feb
  3. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs5 Feb

Other references · 5

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  5. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 20.

26 papers.

Sort:

By area:

  1. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  2. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 40
  3. 2025Base Models Know How to Reason, Thinking Models Learn Whencited by 27
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  5. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  6. 2026Reasoning Models Generate Societies of Thoughtcited by 12
  7. 2025Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectorscited by 8
  8. 2026The Value Axis: Language Models Encode Whether They're on the Right Trackcited by 7
  9. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  10. 2026Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lenscited by 6
  11. 2026How's it going? Reinforcement learning in language models recruits a functional welfare axiscited by 6
  12. 2025Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Processcited by 5
14 more
  1. 2025Rank-1 LoRAs Encode Interpretable Reasoning Signalscited by 4
  2. 2026How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signalscited by 3
  3. 2026Consolidating Rewarded Perturbations for LLM Post-Trainingcited by 3
  4. 2026Convergent World Representations and Divergent Taskscited by 2
  5. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokenscited by 2
  6. 2026Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalcited by 2
  7. 2026Transcoder Adapters for Reasoning-Model Diffingcited by 1
  8. 2026Fluid Reasoning Representationscited by 0
  9. 2026MoRFI: Monotonic Sparse Autoencoder Feature Identificationcited by 0
  10. 2026PathCal: State-Aware Reflection-Marker Calibration for Efficient Reasoningcited by 0
  11. 2026Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMscited by 0
  12. 2026Reasoning Fine-Tuning Induces Persistent Latent Policy Statescited by 0
  13. 2026Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPOcited by 0
  14. 2026The Thinking Leakage in Reinforcement Learning for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
  2. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
  3. 2026The Shape of Overthinking: Backtracking Bursts in Long Reasoning Traces
  4. 2025Internal states before wait modulate reasoning patterns
  5. 2026Emergent Search and Backtracking in Latent Reasoning Models
  6. 2026Decoding the Critique Mechanism in Large Reasoning Models
  7. 2026LogicTrack: Auditing Reasoning Trajectories of Large Language Models with Formal Logic Solvers
  8. 2026Reflection Steering: Disentangling Reflection from Reasoning in Activation Space for Token-Efficient Inference