mlsanityData to 14 Oct 2026

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

arXiv 2502.03699 ↗First posted 6 Feb 2025cs.CL cs.AI cs.IRcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized artificial intelligence with capabilities in reasoning, coding, and communication, driving innovation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing challenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learning (RL)-based alignment methods are notoriously complex, direct optimization approaches offer a simpler alternative. In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles. We present a systematic framework that bridges LLM alignment and IR methodologies, mapping LLM generation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we propose LLM Alignment as Retriever Preference Optimization (LarPO), a new alignment method that enhances overall alignment quality. Extensive experiments validate LarPO's effectiveness with 38.9 % and 13.7 % averaged improvement on AlpacaEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 64 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

51 of 64 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment18 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
31 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023Mistral 7Bcited by 3,164
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  11. 2024A Survey on Large Language Models for Code Generationcited by 658
  12. 2019Passage Re-ranking with BERTcited by 645
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  14. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  16. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 413
  17. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  18. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  19. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  20. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  21. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  22. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  23. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  24. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 155
  25. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  26. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  27. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  28. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  29. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  30. 2022Curriculum Learning for Dense Retrieval Distillationcited by 38
  31. 2024Ranked List Truncation for Large Language Model-based Re-Rankingcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  2. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  3. 2026Distilling the Thought, Watermarking the Answer: A Principle Semantic Guided Watermark for Large Reasoning Modelscited by 3
  4. 2025Test-Time Scaling Strategies for Generative Retrieval in Multimodal Conversational Recommendationscited by 1
  5. 2026S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inferencecited by 1
  6. 2026ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Rerankercited by 0
  7. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2024Sample-Efficient Alignment for LLMs
  3. 2025Alignment of large language models with constrained learning
  4. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  6. 2026EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
  7. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
  8. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment