mlsanityData to 14 Oct 2026

Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

arXiv 2505.19743 ↗First posted 26 May 2025cs.CL cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 38 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

30 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Token-level Direct Preference Optimization18 Apr
  3. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  6. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
11 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  3. 2022Self-critiquing models for assisting human evaluatorscited by 380
  4. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  5. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  6. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  8. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  9. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  10. 2020Dynamic Context Selection for Document-level Neural Machine Translation via Reinforcement Learningcited by 25
  11. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Timecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Training-free Alignment of Large Language Models
  2. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2024LIONs: An Empirically Optimized Approach to Align Language Models
  5. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  6. 2024DeAL: Decoding-time Alignment for Large Language Models
  7. 2026AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
  8. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences