mlsanityData to 7 Oct 2026

Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models

arXiv 2608.00828 ↗First posted 1 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate the geometry of decision-making in Multiple Choice Question Answering (MCQA) through the lens of isotropy. Analyzing five open-weight models across diverse datasets, we identify decision-critical transition layers characterized by a shift in isotropy, coinciding with a major representational change and the emergence of task-relevant clusters. We demonstrate that this synchronized geometric behavior is strongly correlated with downstream accuracy (r≈0.84), displaying its relevance for successful decision-making. Furthermore, we show that this transition is robust to prompt variations, suggesting that it reflects a general mechanism of model behavior.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

32 of 44 references.

References · 32

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  12. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
20 more
  1. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  2. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  3. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  4. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  5. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  6. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  7. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 100
  8. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 92
  9. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  10. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  11. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  12. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  13. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  14. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  15. 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
  16. 2021Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representationscited by 13
  17. 2025Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddingscited by 10
  18. 2025Geometry of Decision Making in Language Modelscited by 8
  19. 2025From Topology to Retrieval: Decoding Embedding Spaces with Unified Signaturescited by 4
  20. 2025Effect of Selection Format on LLM Performancecited by 3

Built on by

Not cited yet.