# The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models

arXiv: https://arxiv.org/abs/2502.08009 · HTML: https://mlsanity.com/paper/2502.08009/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [On the geometry of generalization and memorization in deep neural networks](https://mlsanity.com/paper/2105.14602/) (2021)
- [Classification and Geometry of General Perceptual Manifolds](https://mlsanity.com/paper/1710.06487/) (2017)
- [Neural population geometry: An approach for understanding biological and artificial neural networks](https://mlsanity.com/paper/2104.07059/) (2021)
- [Emergence of Separable Manifolds in Deep Language Representations](https://mlsanity.com/paper/2006.01095/) (2020)
- [Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps](https://mlsanity.com/paper/2307.05052/) (2023)
- [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://mlsanity.com/paper/2303.03307/) (2023)
- [Untangling in Invariant Speech Recognition](https://mlsanity.com/paper/2003.01787/) (2020)
- [Linear Classification of Neural Manifolds with Correlated Variability](https://mlsanity.com/paper/2211.14961/) (2022)

## Built on by (most cited first, 18 of 18)

- [Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning](https://mlsanity.com/paper/2505.18752/) (2025)
- [Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometry](https://mlsanity.com/paper/2503.18114/) (2025)
- [MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper](https://mlsanity.com/paper/2509.00996/) (2025)
- [Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight](https://mlsanity.com/paper/2509.24169/) (2025)
- [Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis](https://mlsanity.com/paper/2509.24164/) (2025)
- [Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings](https://mlsanity.com/paper/2510.19694/) (2025)
- [Diagnosing Generalization Failures from Representational Geometry Markers](https://mlsanity.com/paper/2603.01879/) (2026)
- [Shared Lexical Task Representations Explain Behavioral Variability In LLMs](https://mlsanity.com/paper/2604.22027/) (2026)
- [Large language models reorganize representational geometry during in-context learning](https://mlsanity.com/paper/2605.28854/) (2026)
- [Decomposing how prompting steers behavior](https://mlsanity.com/paper/2606.03093/) (2026)
- [What's in a prompt? Language models encode literary style in prompt embeddings](https://mlsanity.com/paper/2505.17071/) (2025)
- [On the Relationship Between the Choice of Representation and In-Context Learning](https://mlsanity.com/paper/2510.08372/) (2025)
- [Emergent Manifold Separability during Reasoning in Large Language Models](https://mlsanity.com/paper/2602.20338/) (2026)
- [Estimating Dimensionality of Neural Representations from Finite Samples](https://mlsanity.com/paper/2509.26560/) (2025)
- [Optimizing Soft Prompt Tuning via Structural Evolution](https://mlsanity.com/paper/2602.16500/) (2026)
- [From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models](https://mlsanity.com/paper/2609.32653/) (2026)
- [Label Collapse in Diffusion Language Model In-Context Learning](https://mlsanity.com/paper/2610.16839/) (2026)
- [Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs](https://mlsanity.com/paper/2610.18412/) (2026)

## Abstract (arXiv)

Decoder-only language models have the ability to dynamically switch between various computational tasks based on input prompts. Despite many successful applications of prompting, there is very limited understanding of the internal mechanism behind such flexibility. In this work, we investigate how different prompting methods affect the geometry of representations in these models. Employing a framework grounded in statistical physics, we reveal that various prompting techniques, while achieving similar performance, operate through distinct representational mechanisms for task adaptation. Our analysis highlights the critical role of input distribution samples and label semantics in few-shot in-context learning. We also demonstrate evidence of synergistic and interfering interactions between different tasks on the representational level. Our work contributes to the theoretical understanding of large language models and lays the groundwork for developing more effective, representation-aware prompting strategies.