mlsanityData to 7 Oct 2026

Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction

arXiv 2501.05336 ↗First posted 9 Jan 2025cs.CL cs.AI cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The rapid advancement of large language models (LLMs) has led to significant improvements in their capabilities, but also to increased concerns about their alignment with human values and intentions. Current alignment strategies, including adaptive training and inference-time methods, have demonstrated potential in this area. However, these approaches still struggle to balance deployment complexity and capability across various tasks and difficulties. In this work, we introduce the Streaming Distribution Induce Aligner (Stream Aligner), a novel alignment paradigm that combines efficiency with enhanced performance in various tasks throughout the generation process. Stream Aligner achieves dynamic sentence-level correction by using a small model to learn the preferences of the suffix sentence, iteratively correcting the suffix sentence output by the upstream model, and then using the corrected sentence to replace the suffix sentence in subsequent generations. Compared to Aligner, our experiments demonstrate that Stream Aligner reduces reliance on the capabilities of additional models, enhances the reasoning abilities of LLMs, and decreases latency during user interaction. Specifically, Stream Aligner-2B model has achieved an improvement of 76.1% in helpfulness, 36.0% in harmlessness on the tested Llama2-70B-chat model, and Stream Aligner-8B has achieved an improvement of 3.5% on the math ability of the tested Llama3-70B-Instruct model.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 36 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

25 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  2. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  3. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2023Steering Language Models With Activation Engineeringcited by 824
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
10 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  3. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  4. 2022Generating Sequences by Learning to Self-Correctcited by 271
  5. 2023AI Alignment: A Comprehensive Surveycited by 256
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  7. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  8. 2024Tuning Language Models by Proxycited by 88
  9. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  10. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  2. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialoguecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligners: Decoupling LLMs and Alignment
  2. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  3. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  4. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  5. 2023Aligning Large Language Models with Human: A Survey
  6. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  7. 2024Efficient Alignment of Large Language Models via Data Sampling
  8. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning