mlsanityData to 14 Oct 2026

The Value Axis: Language Models Encode Whether They're on the Right Track

arXiv 2606.17056 ↗First posted 15 Jun 2026cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic, in-context reinforcement learning data, we construct a "value" axis for Qwen3-8B. We find that activations along this axis distinguish between high vs. low verbalized confidence, rollouts without and with backtracking, and correct vs. corrupted code. Steering towards high value causally suppresses self-correction and reduces explanatory verbosity, while steering towards low value induces backtracking and exploration. We demonstrate that direct preference optimization (DPO) can increase the internal value of rewarded behaviors (e.g. use a certain word), causing the model to act more confidently after exhibiting them. Finally, we apply the value axis to study in-the-wild settings. For example, we find that Qwen assigns low value to politically sensitive chat queries after post-training and that supervised fine-tuning increases internal confidence within the training domain. Our results suggest that language models linearly encode an estimate of expected goal success that modulates their confidence in pursuing a direction.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

26 of 29 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
14 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  3. 2023Function Vectors in Large Language Modelscited by 308
  4. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  5. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 159
  6. 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 76
  7. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  8. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  9. 2025Steering Evaluation-Aware Language Models to Act Like They Are Deployedcited by 35
  10. 2025Reasoning-Finetuning Repurposes Latent Representations in Base Modelscited by 26
  11. 2026How do LLMs Compute Verbal Confidencecited by 26
  12. 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completioncited by 23
  13. 2025Internal states before wait modulate reasoning patternscited by 6
  14. 2026How's it going? Reinforcement learning in language models recruits a functional welfare axiscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 35
  2. 2026The Halt Vector: Internalizing a Causal Steering Intervention for Efficient Reasoningcited by 0
  3. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  4. 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
  5. 2026Steering Language Model Goals with Value Transplantcited by 0
  6. 2026VERA: Verdict-Conditioned Reliability for Adaptive LLM Judgescited by 0
  7. 2026SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Models Inherit Value Biases from Pretraining
  2. 2025Dual Mechanisms of Value Expression: Intrinsic vs. Prompted Values in Large Language Models
  3. 2024Large Language Models are Biased Reinforcement Learners
  4. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  5. 2024Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective
  6. 2025Alignment is Localized: A Causal Probe into Preference Layers
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  8. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models