# Improving Reasoning Performance in Large Language Models via Representation Engineering arXiv: https://arxiv.org/abs/2504.19483 ยท HTML: https://mlsanity.com/paper/2504.19483/ ## Key facts - Cited by 23 papers in the map (data through 14 Oct 2026). - Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. - Builds on 13 of its 22 references that are in the map. - First posted on arXiv 28 Apr 2025. ## Placed in - LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/) - Mechanistic interpretability / [Steering and interpreting LLM reasoning](https://mlsanity.com/mechinterp/steering-and-interpreting-llm-reasoning/) ## Builds on (most cited first, 13 of 13) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016) - [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023) - [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022) - [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022) - [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023) - [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023) - [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023) - [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023) - [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024) - [Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning](https://mlsanity.com/paper/2402.14856/) (2024) ## Built on by (most cited first, 23 of 23) - [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026) - [Large Language Models Encode Semantics and Alignment in Linearly Separable Representations](https://mlsanity.com/paper/2507.09709/) (2025) - [Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2505.15634/) (2025) - [EasySteer: A Unified Framework for High-Performance and Extensible LLM Steering](https://mlsanity.com/paper/2509.25175/) (2025) - [Meaningless Tokens, Meaningful Gains: How Activation Shifts Enhance LLM Reasoning](https://mlsanity.com/paper/2510.01032/) (2025) - [Enhancing Cross-task Transfer of Large Language Models via Activation Steering](https://mlsanity.com/paper/2507.13236/) (2025) - [Using LLMs for Late Multimodal Sensor Fusion for Activity Recognition](https://mlsanity.com/paper/2509.10729/) (2025) - [Beyond representational alignment with brain-guided language models for robust reasoning](https://mlsanity.com/paper/2606.11893/) (2026) - [ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations](https://mlsanity.com/paper/2510.27355/) (2025) - [ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval](https://mlsanity.com/paper/2602.01639/) (2026) - [Explaining Intrinsic Moral Self-Correction with Mechanistic Interpretability](https://mlsanity.com/paper/2505.11924/) (2025) - [Unveiling the Latent Directions of Reflection in Large Language Models](https://mlsanity.com/paper/2508.16989/) (2025) - [Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization](https://mlsanity.com/paper/2511.19131/) (2025) - [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026) - [Bias-Restrained Prefix Representation Finetuning for Mathematical Reasoning](https://mlsanity.com/paper/2511.10707/) (2025) - [Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learning](https://mlsanity.com/paper/2601.07903/) (2026) - [Tracing Uncertainty in Language Model "Reasoning"](https://mlsanity.com/paper/2605.07776/) (2026) - [DenseSteer: Steering Small Language Models towards Dense Math Reasoning](https://mlsanity.com/paper/2605.29247/) (2026) - [Polished but Unresolved: Identifying Late-Stage Pressure States in Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2609.00823/) (2026) - [Continuous Interpretive Steering for Scalar Diversity](https://mlsanity.com/paper/2604.07006/) (2026) - [Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance](https://mlsanity.com/paper/2609.24066/) (2026) - [Query-Adaptive Neuron Steering for LLMs](https://mlsanity.com/paper/2610.14519/) (2026) - [SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning](https://mlsanity.com/paper/2610.16104/) (2026) ## Abstract (arXiv) Recent advancements in large language models (LLMs) have resulted in increasingly anthropomorphic language concerning the ability of LLMs to reason. Whether reasoning in LLMs should be understood to be inherently different is, however, widely debated. We propose utilizing a representation engineering approach wherein model activations are read from the residual stream of an LLM when processing a reasoning task. The activations are used to derive a control vector that is applied to the model as an inference-time intervention, modulating the representational space of the model, to improve performance on the specified task. We publish the code for deriving control vectors and analyzing model representations. The method allows us to improve performance on reasoning benchmarks and assess how control vectors influence the final logit distribution of a model via metrics such as KL divergence and entropy. We apply control vectors to Mistral-7B-Instruct and a range of Pythia models on an inductive, a deductive and mathematical reasoning task. We show that an LLM can, to a certain degree, be controlled to improve its perceived reasoning ability by modulating activations. The intervention is dependent upon the ability to reliably extract the model's typical state when correctly solving a task. Our results suggest that reasoning performance can be modulated in the same manner as other information-processing tasks performed by LLMs and demonstrate that we are capable of improving performance on specific tasks via a simple intervention on the residual stream with no additional training.