mlsanityData to 14 Oct 2026

Self-Improving Model Steering

arXiv 2507.08967 ↗First posted 11 Jul 2025cs.CLcited by 3

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Model steering represents a powerful technique that dynamically aligns large language models (LLMs) with human preferences during inference. However, conventional model-steering methods rely heavily on externally annotated data, not only limiting their adaptability to varying contexts but also tethering their effectiveness to annotation quality. In this paper, we present SIMS, the first self-improving model-steering framework that operates without relying on external supervision. At its core, SIMS autonomously generates and refines contrastive samples through iterative self-improvement cycles, enabling adaptive, context-specific steering. Additionally, SIMS employs novel strategies, including prompt ranking and contrast sampling, to further enhance steering efficacy. Extensive evaluation across diverse LLMs and benchmarks demonstrates that SIMS substantially outperforms existing methods in steering effectiveness and adaptability, highlighting self-improving model steering as a promising direction for future research on inference-time LLM alignment.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 41 of its 49 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

41 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Self-Evolved Reward Learning for LLMs1 Nov
  2. 2024Progress or Regress? Self-Improvement Reversal in Post-training6 Jul
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
26 more
  1. 2023Steering Language Models With Activation Engineeringcited by 922
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  4. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  7. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  8. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  9. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  10. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  11. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  12. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 95
  13. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  14. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  15. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  16. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 68
  17. 2024Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectorscited by 63
  18. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 56
  19. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  20. 2025LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimizationcited by 24
  21. 2024Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspectivecited by 23
  22. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  23. 2024Context Steering: Controllable Personalization at Inference Timecited by 22
  24. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 21
  25. 2024Self-Improving Robust Preference Optimizationcited by 19
  26. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 13
  2. 2025Efficient Cold-Start Recommendation via BPE Token-Level Embedding Initialization with LLMcited by 2
  3. 2026Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Where to Steer: Input-Dependent Layer Selection for Steering Improves LLM Alignment
  2. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtracking
  3. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics
  4. 2025On the Limitations of Steering in Language Model Alignment
  5. 2025Improved Representation Steering for Language Models
  6. 2025Effectively Steer LLM To Follow Preference via Building Confident Directions
  7. 2026Probabilistic Concept-Aware Steering for Trustworthy LLM Inference
  8. 2026FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models