# How's it going? Reinforcement learning in language models recruits a functional welfare axis

arXiv: https://arxiv.org/abs/2605.30232 · HTML: https://mlsanity.com/paper/2605.30232/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 11 of 11)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://mlsanity.com/paper/2509.07968/) (2025)
- [Reasoning-Finetuning Repurposes Latent Representations in Base Models](https://mlsanity.com/paper/2507.12638/) (2025)
- [There Is More to Refusal in Large Language Models than a Single Direction](https://mlsanity.com/paper/2602.02132/) (2026)
- [Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis](https://mlsanity.com/paper/2506.04574/) (2025)
- [A Unified Representation Underlying the Judgment of Large Language Models](https://mlsanity.com/paper/2510.27328/) (2025)

## Built on by (most cited first, 6 of 6)

- [The Value Axis: Language Models Encode Whether They're on the Right Track](https://mlsanity.com/paper/2606.17056/) (2026)
- [The Halt Vector: Internalizing a Causal Steering Intervention for Efficient Reasoning](https://mlsanity.com/paper/2608.28859/) (2026)
- [Replicating the Geometry of Emotion Representations in a Base Open-Weights Model](https://mlsanity.com/paper/2609.22208/) (2026)
- [Same Outcome, Different Readout: What Does a Steerable Valence Direction in LLMs Represent?](https://mlsanity.com/paper/2609.22850/) (2026)
- [Steering Language Model Goals with Value Transplant](https://mlsanity.com/paper/2609.34056/) (2026)
- [Language Models Act on Hidden Valence](https://mlsanity.com/paper/2609.35591/) (2026)

## Abstract (arXiv)

How does reinforcement learning shape a language model's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.