# Controlling Large Language Models Through Concept Activation Vectors

arXiv: https://arxiv.org/abs/2501.05764 · HTML: https://mlsanity.com/paper/2501.05764/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2025.

## Built on by (most cited first, 28 of 28)

- [Internal Value Alignment in Large Language Models through Controlled Value Vector Activation](https://mlsanity.com/paper/2507.11316/) (2025)
- [SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models](https://mlsanity.com/paper/2510.26769/) (2025)
- [Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMs](https://mlsanity.com/paper/2505.15524/) (2025)
- [Do LLMs Strategically Reveal, Conceal, and Infer Information? A Theoretical and Empirical Analysis in The Chameleon Game](https://mlsanity.com/paper/2501.19398/) (2025)
- [Probing the Robustness of Large Language Models Safety to Latent Perturbations](https://mlsanity.com/paper/2506.16078/) (2025)
- [Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning](https://mlsanity.com/paper/2604.12479/) (2026)
- [LLM Explainability with Counterfactual Chains and Causal Graphs](https://mlsanity.com/paper/2606.05972/) (2026)
- [VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models](https://mlsanity.com/paper/2509.25533/) (2025)
- [Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions](https://mlsanity.com/paper/2601.04465/) (2026)
- [VISPA: Pluralistic Alignment via Automatic Value Selection and Activation](https://mlsanity.com/paper/2601.12758/) (2026)
- [Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs](https://mlsanity.com/paper/2601.18483/) (2026)
- [RAPTOR: Ridge-Adaptive Logistic Probes](https://mlsanity.com/paper/2602.00158/) (2026)
- [FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance](https://mlsanity.com/paper/2602.02060/) (2026)
- [Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads](https://mlsanity.com/paper/2603.05772/) (2026)
- [On the Limits of Steering Vectors for Preference-Aligned Generation](https://mlsanity.com/paper/2607.01802/) (2026)
- [When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems](https://mlsanity.com/paper/2607.06807/) (2026)
- [Out-of-Distribution Detection via Channelwise Feature Aggregation in Neural Network-Based Receivers](https://mlsanity.com/paper/2505.15570/) (2025)
- [VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models](https://mlsanity.com/paper/2508.08521/) (2025)
- [Keep Calm and Avoid Harmful Content: Concept Alignment and Latent Manipulation Towards Safer Answers](https://mlsanity.com/paper/2510.12672/) (2025)
- [On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception](https://mlsanity.com/paper/2511.06519/) (2025)
- [Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models](https://mlsanity.com/paper/2602.02304/) (2026)
- [Conversational Control with Ontologies for Large Language Models: A Lightweight Framework for Constrained Generation](https://mlsanity.com/paper/2604.04450/) (2026)
- [Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs](https://mlsanity.com/paper/2604.08846/) (2026)
- [α-TCAV: A Unified Framework for Testing with Concept Activation Vectors](https://mlsanity.com/paper/2605.15688/) (2026)
- [Latent Fact-Checking: Detecting Misinformation through Activation Engineering](https://mlsanity.com/paper/2608.06417/) (2026)
- [ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on](https://mlsanity.com/paper/2608.26083/) (2026)
- [GUIDE: Guiding Internal Evidence with Language Instructions](https://mlsanity.com/paper/2608.30712/) (2026)
- [Predicting Steering Vectors and Adapter Weights for Few-Shot Author-Style Transfer](https://mlsanity.com/paper/2610.03163/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are widely deployed across various domains, the ability to control their generated outputs has become more critical. This control involves aligning LLMs outputs with human values and ethical principles or customizing LLMs on specific topics or styles for individual users. Existing controlled generation methods either require significant computational resources and extensive trial-and-error or provide coarse-grained control. In this paper, we propose Generation with Concept Activation Vector (GCAV), a lightweight model control framework that ensures accurate control without requiring resource-extensive fine-tuning. Specifically, GCAV first trains a concept activation vector for specified concepts to be controlled, such as toxicity. During inference, GCAV steers the concept vector in LLMs, for example, by removing the toxicity concept vector from the activation layers. Control experiments from different perspectives, including toxicity reduction, sentiment control, linguistic style, and topic control, demonstrate that our framework achieves state-of-the-art performance with granular control, allowing for fine-grained adjustments of both the steering layers and the steering magnitudes for individual samples.