mlsanityData to 14 Oct 2026

Beyond Fixed Directions: Adaptive Representation Analysis of Reasoning and Memorization in LLMs

arXiv 2608.21919 ↗First posted 22 Aug 2026cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent work has proposed that reasoning and memorization in language models can be characterized by a single representation direction, including methods that keep this direction fixed during reinforcement learning. We test two assumptions behind this view. First, are reasoning-oriented and factual-recall task groups approximately single-direction separable? Second, does the resulting geometry remain stable after GRPO? Using Qwen3-0.6B and a controlled 400-example dataset, we find that a one-dimensional projection can match a full 1024-dimensional linear probe with AUROC = 1.00 on the studied task groups. However, after GRPO, the corresponding direction is substantially reorganized: mean-direction cosine averages 0.453, probe-direction cosine 0.445, while direct representation drift reaches 0.511 at the final layer. Probe AUROC nevertheless remains 1.00. The evidence therefore supports single-direction decodability for the studied task groups but challenges fixed-direction stability: the information persists while its geometric realization changes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 21 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

15 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction29 Mar
  2. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models23 Oct

Other references · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  12. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
1 more
  1. 2026Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
  2. 2025Forgetting as a Feature: Cognitive Alignment of Large Language Models
  3. 2026LLM Reasoning Is Latent, Not the Chain of Thought
  4. 2026Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
  5. 2026Abstract representational geometry supports inference in large language models
  6. 2025Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis
  7. 2025On the Failure of Latent State Persistence in Large Language Models
  8. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals