# Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization

arXiv: https://arxiv.org/abs/2511.19131 · HTML: https://mlsanity.com/paper/2511.19131/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 28 of 28)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Chain-of-Thought Reasoning Without Prompting](https://mlsanity.com/paper/2402.10200/) (2024)
- [Programming Refusal with Conditional Activation Steering](https://mlsanity.com/paper/2409.05907/) (2024)
- [Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2406.09136/) (2024)
- [CoT-Kinetics: A Theoretical Modeling Assessing LRM Reasoning Process](https://mlsanity.com/paper/2505.13408/) (2025)
- [A Language Model's Guide Through Latent Space](https://mlsanity.com/paper/2402.14433/) (2024)
- [Personality Alignment of Large Language Models](https://mlsanity.com/paper/2408.11779/) (2024)
- [Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering](https://mlsanity.com/paper/2503.11314/) (2025)
- [Steering off Course: Reliability Challenges in Steering Language Models](https://mlsanity.com/paper/2504.04635/) (2025)
- [Linear Representations of Political Perspective Emerge in Large Language Models](https://mlsanity.com/paper/2503.02080/) (2025)
- [Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution](https://mlsanity.com/paper/2410.00153/) (2024)
- [Improving Reasoning Performance in Large Language Models via Representation Engineering](https://mlsanity.com/paper/2504.19483/) (2025)
- [The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2503.23084/) (2025)
- [JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulation](https://mlsanity.com/paper/2502.20684/) (2025)
- [ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLM Intrinsic Reasoning](https://mlsanity.com/paper/2504.06650/) (2025)

## Built on by (most cited first, 2 of 2)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models](https://mlsanity.com/paper/2601.14270/) (2026)

## Abstract (arXiv)

Chain-of-Thought (CoT) reasoning is a critical capability for large language models (LLMs), enabling them to tackle com- plex multi-step tasks. While base LLMs, pre-trained on general text corpora, often struggle with reasoning due to a lack of specialized training, recent studies reveal their latent reason- ing potential tied to hidden states. However, existing hidden state manipulation methods, such as linear activation steering, suffer from limitations due to their rigid and unconstrained nature, often leading to distribution shifts and degraded text quality. In this work, we propose a novel approach for elic- iting CoT reasoning from base LLMs through hidden state manipulation grounded in probabilistic conditional generation. By reformulating the challenge as an optimization problem with a balanced likelihood and prior regularization framework, our method guides hidden states toward reasoning-oriented trajectories while preserving linguistic coherence. Extensive evaluations across mathematical, commonsense, and logical reasoning benchmarks demonstrate that our approach con- sistently outperforms existing steering methods, offering a theoretically principled and effective solution for enhancing reasoning capabilities in base LLMs.