mlsanityData to 7 Oct 2026

The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models

arXiv 2502.08009 ↗First posted 11 Feb 2025cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Decoder-only language models have the ability to dynamically switch between various computational tasks based on input prompts. Despite many successful applications of prompting, there is very limited understanding of the internal mechanism behind such flexibility. In this work, we investigate how different prompting methods affect the geometry of representations in these models. Employing a framework grounded in statistical physics, we reveal that various prompting techniques, while achieving similar performance, operate through distinct representational mechanisms for task adaptation. Our analysis highlights the critical role of input distribution samples and label semantics in few-shot in-context learning. We also demonstrate evidence of synergistic and interfering interactions between different tasks on the representational level. Our work contributes to the theoretical understanding of large language models and lays the groundwork for developing more effective, representation-aware prompting strategies.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

30 of 47 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  12. 2022Transformers learn in-context by gradient descentcited by 714
18 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2015Generating Sentences from a Continuous Spacecited by 607
  3. 2023In-Context Learning Creates Task Vectorscited by 302
  4. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  5. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  6. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  7. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  8. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  9. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  10. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  11. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  12. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  13. 2021Neural population geometry: An approach for understanding biological and artificial neural networkscited by 52
  14. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 39
  15. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Mapscited by 35
  16. 2023Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representationscited by 31
  17. 2020Untangling in Invariant Speech Recognitioncited by 12
  18. 2022Linear Classification of Neural Manifolds with Correlated Variabilitycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  2. 2025Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometrycited by 9
  3. 2025MEPT: Mixture of Expert Prompt Tuning as a Manifold Mappercited by 5
  4. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  5. 2026Diagnosing Generalization Failures from Representational Geometry Markerscited by 2
  6. 2026Large language models reorganize representational geometry during in-context learningcited by 2
  7. 2025On the Relationship Between the Choice of Representation and In-Context Learningcited by 1
  8. 2025Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddingscited by 1
  9. 2025What's in a prompt? Language models encode literary style in prompt embeddingscited by 0
  10. 2025Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysiscited by 0
  11. 2025Estimating Dimensionality of Neural Representations from Finite Samplescited by 0
  12. 2026Optimizing Soft Prompt Tuning via Structural Evolutioncited by 0
2 more
  1. 2026Emergent Manifold Separability during Reasoning in Large Language Modelscited by 0
  2. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Modelscited by 0