# Relational Preference Encoding in Looped Transformer Internal States

arXiv: https://arxiv.org/abs/2604.09870 · HTML: https://mlsanity.com/paper/2604.09870/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 5 of its 12 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 5 of 5)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Emotion Concepts and their Function in a Large Language Model](https://mlsanity.com/paper/2604.07729/) (2026)

## Built on by (most cited first, 2 of 2)

- [Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning](https://mlsanity.com/paper/2605.21488/) (2026)
- [Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary](https://mlsanity.com/paper/2607.18553/) (2026)

## Abstract (arXiv)

We investigate how looped transformers encode human preference, training lightweight evaluator heads on frozen Ouro-2.6B loop-iteration states on Anthropic HH-RLHF. v2: an erratum is prepended; the original manuscript is unchanged. A post-publication audit found the three headline results inflated by two independent evaluation errors. The 95.2% pairwise evaluator accuracy is a canonical-ordering artifact: the data were correctly split, but the evaluator learned to prefer the first-presented argument; its strict antisymmetrized accuracy on the full 8,552-pair test set is 63.9%. The 84.5% pairwise probe and the below-chance 21.75% pointwise probe were source-item leaks (orientation rows and pair partners crossing the train/test split); corrected pair-disjoint values are 56.5% and 54.2% -- above chance, so the "inverted polarity" finding is withdrawn. The central finding survives at much smaller magnitude: preference is decoded more accurately relationally than pointwise (paired +2.3 points, 95% CI [+1.3, +3.3]), and the antisymmetrized evaluator still beats the linear probe, but no corrected readout rivals end-to-end reward models. The methodological findings stand (constant-output degeneracy, flip test, swap-protocol metric deflation), with one correction: antisymmetrized accuracy, not antisymmetry correlation, certifies relational discrimination. The two errors are mutually invisible -- a split audit cannot see an ordering prior, antisymmetrization cannot see a leak -- so both checks are required. Full audit in the follow-up work (Kirin, 2026, in preparation).