mlsanityData to 14 Oct 2026

DenseSteer: Steering Small Language Models towards Dense Math Reasoning

arXiv 2605.29247 ↗First posted 28 May 2026cs.AI cs.CL cs.LGcited by 1

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) demonstrate strong chain-of-thought (CoT) reasoning abilities, while smaller models (<= 3B parameters) significantly underperform on multi-step reasoning tasks. Based on empirical analyses of the Qwen-2.5 model family on math reasoning benchmarks, we find that more proficient reasoning is associated with fewer reasoning steps but higher information density per step, a property we term Dense Reasoning. Motivated by this observation, we propose DenseSteer, a training-free inference-time steering framework that enhances small-model reasoning by modulating internal representations toward dense reasoning patterns. Experiments show that our method yields consistent accuracy improvements without increasing token-level Negative Log-Likelihood, highlighting dense reasoning as an effective structural approach to mathematical problem solving.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 32 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

23 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math30 Apr
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
9 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  2. 2023Steering Language Models With Activation Engineeringcited by 922
  3. 2024DeepSeek-V3 Technical Reportcited by 811
  4. 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
  5. 2020Densely Guided Knowledge Distillation using Multiple Teacher Assistantscited by 83
  6. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 69
  7. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 68
  8. 2025Improving Reasoning Performance in Large Language Models via Representation Engineeringcited by 23
  9. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Closing the Prior-Posterior Loop: Self-Reflective Molecular Design with Analysis-Driven LLM Iterationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Large Language Models for Mathematical Reasoning
  2. 2024Step Guided Reasoning: Improving Mathematical Reasoning using Guidance Generation and Step Reasoning
  3. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  4. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task
  5. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  6. 2026The Missing Primitive: Diagnosing and Repairing Mathematical Reasoning in Large Language Models
  7. 2024Small Language Models are Equation Reasoners
  8. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks