mlsanityData to 14 Oct 2026

Textual Self-attention Network: Test-Time Preference Optimization through Textual Gradient-based Attention

arXiv 2511.06682 ↗First posted 10 Nov 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable generalization capabilities, but aligning their outputs with human preferences typically requires expensive supervised fine-tuning. Recent test-time methods leverage textual feedback to overcome this, but they often critique and revise a single candidate response, lacking a principled mechanism to systematically analyze, weigh, and synthesize the strengths of multiple promising candidates. Such a mechanism is crucial because different responses may excel in distinct aspects (e.g., clarity, factual accuracy, or tone), and combining their best elements may produce a far superior outcome. This paper proposes the Textual Self-Attention Network (TSAN), a new paradigm for test-time preference optimization that requires no parameter updates. TSAN emulates self-attention entirely in natural language to overcome this gap: it analyzes multiple candidates by formatting them into textual keys and values, weighs their relevance using an LLM-based attention module, and synthesizes their strengths into a new, preference-aligned response under the guidance of the learned textual attention. This entire process operates in a textual gradient space, enabling iterative and interpretable optimization. Empirical evaluations demonstrate that with just three test-time iterations on a base SFT model, TSAN outperforms supervised models like Llama-3.1-70B-Instruct and surpasses the current state-of-the-art test-time alignment method by effectively leveraging multiple candidate solutions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 36 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

23 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback22 Jan
  2. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  5. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language24 Nov
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  11. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
  12. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 92
5 more
  1. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  2. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  3. 2024A Survey on LLM Inference-Time Self-Improvementcited by 17
  4. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  5. 2025Training Language Model to Critique for Better Refinementcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024TSO: Self-Training with Scaled Preference Optimization
  2. 2026Token-weighted Direct Preference Optimization with Attention
  3. 2023LLM-augmented Preference Learning from Natural Language
  4. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
  5. 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes
  6. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
  7. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  8. 2025Learning from Self Critique and Refinement for Faithful LLM Summarization