mlsanityData to 7 Oct 2026

Latent Distance Guided Alignment Training for Large Language Models

arXiv 2404.06390 ↗First posted 9 Apr 2024cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Ensuring alignment with human preferences is a crucial characteristic of large language models (LLMs). Presently, the primary alignment methods, RLHF and DPO, require extensive human annotation, which is expensive despite their efficacy. The significant expenses associated with current alignment techniques motivate researchers to investigate the development of annotation-free alignment training methods. In pursuit of improved alignment without relying on external annotation, we introduce Latent Distance Guided Alignment Training (LD-Align). This approach seeks to align the model with a high-quality supervised fine-tune dataset using guidance from a latent space. The latent space is generated through sample reconstruction, akin to auto-encoding. Consequently, we utilize the distance between sample pairs in the latent space to guide DPO-based alignment training. Extensive experimentation and evaluation show the efficacy of our proposed method in achieving notable alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 16 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

13 of 16 references.

References · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
1 more
  1. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Training-free Alignment of Large Language Models
  2. 2024Efficient Alignment of Large Language Models via Data Sampling
  3. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples
  4. 2023Aligning Large Language Models with Human: A Survey
  5. 2024Towards Scalable Automated Alignment of LLMs: A Survey
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment
  8. 2024Hybrid Alignment Training for Large Language Models