# Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks

arXiv: https://arxiv.org/abs/2411.07213 · HTML: https://mlsanity.com/paper/2411.07213/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [Interpreting the Second-Order Effects of Neurons in CLIP](https://mlsanity.com/paper/2406.04341/) (2024)

## Built on by (most cited first, 14 of 14)

- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Understanding (Un)Reliability of Steering Vectors in Language Models](https://mlsanity.com/paper/2505.22637/) (2025)
- [Steering off Course: Reliability Challenges in Steering Language Models](https://mlsanity.com/paper/2504.04635/) (2025)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectives](https://mlsanity.com/paper/2511.06626/) (2025)
- [Analysing the Safety Pitfalls of Steering Vectors](https://mlsanity.com/paper/2603.24543/) (2026)
- [Causality neq Invariance: Function and Concept Vectors in LLMs](https://mlsanity.com/paper/2602.22424/) (2026)
- [What Can We Actually Steer? A Multi-Behavior Study of Activation Control](https://mlsanity.com/paper/2511.18284/) (2025)
- [The Cylindrical Representation Hypothesis for Language Model Steering](https://mlsanity.com/paper/2605.01844/) (2026)
- [But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors](https://mlsanity.com/paper/2505.17760/) (2025)
- [Transmuting prompts into weights](https://mlsanity.com/paper/2510.08734/) (2025)
- [A Low-Rank Subspace Analysis of LLM Interventions](https://mlsanity.com/paper/2606.14388/) (2026)
- [Beyond Multiple Choice: Evaluating Steering Vectors for Summarization](https://mlsanity.com/paper/2505.24859/) (2025)
- [Detecting and Controlling Sycophancy with Cascading Linear Features](https://mlsanity.com/paper/2606.26155/) (2026)

## Abstract (arXiv)

A key objective of interpretability research on large language models (LLMs) is to develop methods for robustly steering models toward desired behaviors. To this end, two distinct approaches to interpretability -- ``bottom-up" and ``top-down" -- have been presented, but there has been little quantitative comparison between them. We present a case study comparing the effectiveness of representative vector steering methods from each branch: function vectors (FV; arXiv:2310.15213), as a bottom-up method, and in-context vectors (ICV; arXiv:2311.06668) as a top-down method. While both aim to capture compact representations of broad in-context learning tasks, we find they are effective only on specific types of tasks: ICVs outperform FVs in behavioral shifting, whereas FVs excel in tasks requiring more precision. We discuss the implications for future evaluations of steering methods and for further research into top-down and bottom-up steering given these findings.