# Concept Heterogeneity-aware Representation Steering arXiv: https://arxiv.org/abs/2603.02237 ยท HTML: https://mlsanity.com/paper/2603.02237/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. - Builds on 32 of its 46 references that are in the map. - First posted on arXiv 13 Feb 2026. ## Placed in - Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/) ## Builds on (most cited first, 30 of 32) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013) - [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016) - [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023) - [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023) - [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024) - [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023) - [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023) - [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023) - [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015) - [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023) - [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024) - [Gromov-Wasserstein Alignment of Word Embedding Spaces](https://mlsanity.com/paper/1809.00013/) (2018) - [Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories](https://mlsanity.com/paper/2406.00034/) (2024) - [A Wasserstein-type distance in the space of Gaussian Mixture Models](https://mlsanity.com/paper/1907.05254/) (2019) - [Improving Activation Steering in Language Models with Mean-Centring](https://mlsanity.com/paper/2312.03813/) (2023) - [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024) - [Attenuating Bias in Word Vectors](https://mlsanity.com/paper/1901.07656/) (2019) - [Rates of Estimation of Optimal Transport Maps using Plug-in Estimators via Barycentric Projections](https://mlsanity.com/paper/2107.01718/) (2021) - [Angular Steering: Behavior Control via Rotation in Activation Space](https://mlsanity.com/paper/2510.26243/) (2025) - [Representation Surgery: Theory and Practice of Affine Steering](https://mlsanity.com/paper/2402.09631/) (2024) - [Statistical optimal transport](https://mlsanity.com/paper/2407.18163/) (2024) - [Hierarchical Optimal Transport for Multimodal Distribution Alignment](https://mlsanity.com/paper/1906.11768/) (2019) - [Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models](https://mlsanity.com/paper/2412.14528/) (2024) - [Mitigating Overthinking in Large Reasoning Models via Manifold Steering](https://mlsanity.com/paper/2505.22411/) (2025) - [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021) ## Built on by (most cited first, 6 of 6) - [Decomposing how prompting steers behavior](https://mlsanity.com/paper/2606.03093/) (2026) - [Metacognitive Steering: Learning the Structure of Scientific Judgment](https://mlsanity.com/paper/2609.16245/) (2026) - [Minimally Invasive Steering of Language Models](https://mlsanity.com/paper/2609.30218/) (2026) - [Kernelized Activation Steering](https://mlsanity.com/paper/2610.01062/) (2026) - [One Token Can Be Enough: Bridging Prompting and Activation Steering with Prefix Steering](https://mlsanity.com/paper/2610.04967/) (2026) - [Steering by Influence: Curvature Aware Data Weighting for Activation Steering](https://mlsanity.com/paper/2610.06383/) (2026) ## Abstract (arXiv) Representation steering offers a lightweight mechanism for controlling the behavior of large language models (LLMs) by intervening on internal activations at inference time. Most existing methods rely on a single global steering direction, typically obtained via difference-in-means over contrastive datasets. This approach implicitly assumes that the target concept is homogeneously represented across the embedding space. In practice, however, LLM representations can be highly non-homogeneous, exhibiting clustered, context-dependent structure, which renders global steering directions brittle. In this work, we view representation steering through the lens of optimal transport (OT), noting that standard difference-in-means steering implicitly corresponds to the OT map between two identical distributions with differing first moments, yielding a global translation. To relax this restrictive assumption, we theoretically model source and target representations as Gaussian mixture models and formulate steering as a discrete OT problem between semantic latent clusters. From the resulting transport plan, we derive an explicit, input-dependent steering map via barycentric projection, producing a smooth, kernel-weighted combination of cluster-level shifts. We term this method Concept Heterogeneity-aware Representation Steering (CHaRS). Through numerous experimental settings, we show that CHaRS yields more effective behavioral control than global steering.