# Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden States

arXiv: https://arxiv.org/abs/2605.17144 · HTML: https://mlsanity.com/paper/2605.17144/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 16 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)
- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Mechanistic interpretability / [Steering and probing for LLM safety](https://mlsanity.com/mechinterp/steering-and-probing-for-llm-safety/)

## Builds on (most cited first, 29 of 29)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model](https://mlsanity.com/paper/2412.13630/) (2024)
- [Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering](https://mlsanity.com/paper/2410.16314/) (2024)
- [Mechanistic interpretability for steering vision-language-action models](https://mlsanity.com/paper/2509.00328/) (2025)
- [Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification](https://mlsanity.com/paper/2510.16281/) (2025)
- [Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach](https://mlsanity.com/paper/2512.02834/) (2025)
- [Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control](https://mlsanity.com/paper/2602.13193/) (2026)
- [Contrastive Representation Regularization for Vision-Language-Action Models](https://mlsanity.com/paper/2510.01711/) (2025)
- [Controlling Recurrent Neural Networks by Conceptors](https://mlsanity.com/paper/1403.3369/) (2014)
- [Reinforcement Learning for Flow-Matching Policies](https://mlsanity.com/paper/2507.15073/) (2025)
- [Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations](https://mlsanity.com/paper/2511.22697/) (2025)
- [Learning Affordances at Inference-Time for Vision-Language-Action Models](https://mlsanity.com/paper/2510.19752/) (2025)
- [OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies](https://mlsanity.com/paper/2603.10052/) (2026)
- [Latent Activation Editing: Inference-Time Refinement of Learned Policies for Safer Multirobot Navigation](https://mlsanity.com/paper/2509.20623/) (2025)
- [Conceptors for Semantic Steering](https://mlsanity.com/paper/2605.04980/) (2026)
- [Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steering](https://mlsanity.com/paper/2602.06022/) (2026)

## Built on by (most cited first, 3 of 3)

- [Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control](https://mlsanity.com/paper/2607.14943/) (2026)
- [Policy Gradient Steering: Interventions from Behavioral Objectives](https://mlsanity.com/paper/2607.27574/) (2026)
- [The Linear Representation Hypothesis for Vision-Language-Action Models](https://mlsanity.com/paper/2609.30996/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models leverage powerful perceptual priors from web-scale Vision-Language Model (VLM) pre-training, yet they remain surprisingly brittle in practice, frequently failing at simple robotic tasks. To mitigate this, we propose Contrastive Conceptor Activation Steering (COAST). COAST builds on the notion of a "conceptor", a linear operator that soft-projects data into the principal components of a target distribution. COAST uses conceptors to identify success-critical subspaces for a target robotic task from a few examples of success and failure rollouts. At inference time, it steers VLA latents into these identified success subspaces to improve task outcomes. Across three architecturally distinct neural policies (flow-matching VLA, autoregressive VLA, and Diffusion Policy), COAST improves absolute mean simulation and real-robot task success rate by over 20 and 40% respectively. The activation subspace geometry reveals that failure modes share substantial structure across tasks while success representations remain largely task-specific. When tasks share similar failure modes, this structure enables previously fitted conceptors to improve performance on new tasks without refitting. Ultimately, our results suggest that current VLAs retain substantial task-relevant knowledge in their latent representations, and that the action expert's decoding bottleneck could be mitigated by steering its residual stream toward task-relevant subspaces. COAST provides a lightweight, training-free path to unlocking these latent capabilities by steering the model towards its own "success" distributions.