mlsanityData to 14 Oct 2026

LoRi: Low-Rank Distillation for Implicit Reasoning

arXiv 2606.05315 ↗First posted 3 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 34 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

18 of 34 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Triggering Chain-of-Thought via Latent Feature Interventions in Large Language Models12 Jan
  2. 2025SIM-CoT: Supervised Implicit Chain-of-Thought24 Sep
  3. 2025Parallel Continuous Chain-of-Thought with Jacobi Iteration23 Jun
  4. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation28 Feb
  5. 2024Training Large Language Models to Reason in a Continuous Latent Space9 Dec
  6. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  7. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step23 May
  8. 2023Implicit Chain of Thought Reasoning via Knowledge Distillation2 Nov
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2025The Origins of Representation Manifolds in Large Language Modelscited by 51
  5. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  6. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 24
  7. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 9
  8. 2025KaVa: Latent Reasoning via Compressed KV-Cache Distillationcited by 8
  9. 2025REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Modelcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning
  2. 2026MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation
  3. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
  4. 2026CIRF: Tokenizing Chain-of-Thoughts into Reusable Functional Units for Efficient Latent Reasoning in Large Language Models
  5. 2026Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
  6. 2025SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
  7. 2022Distilling Reasoning Capabilities into Smaller Language Models
  8. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks