# PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

arXiv: https://arxiv.org/abs/2412.14510 · HTML: https://mlsanity.com/paper/2412.14510/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)
- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 16 of 16)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training](https://mlsanity.com/paper/2405.20978/) (2024)
- [Training Language Models to Generate Text with Citations via Fine-grained Rewards](https://mlsanity.com/paper/2402.04315/) (2024)
- [ATM: Adversarial Tuning Multi-agent System Makes a Robust Retrieval-Augmented Generator](https://mlsanity.com/paper/2405.18111/) (2024)
- [Unsupervised Large Language Model Alignment for Information Retrieval via Contrastive Feedback](https://mlsanity.com/paper/2309.17078/) (2023)

## Built on by (most cited first, 11 of 11)

- [Towards AI Search Paradigm](https://mlsanity.com/paper/2506.17188/) (2025)
- [EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Models](https://mlsanity.com/paper/2603.19532/) (2026)
- [LLP: LLM-Based Product Pricing in E-commerce](https://mlsanity.com/paper/2510.09347/) (2025)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [Human Cognition Inspired RAG with Knowledge Graph for Complex Problem Solving](https://mlsanity.com/paper/2503.06567/) (2025)
- [CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models](https://mlsanity.com/paper/2603.04406/) (2026)
- [RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models](https://mlsanity.com/paper/2506.02726/) (2025)
- [Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA](https://mlsanity.com/paper/2603.24580/) (2026)
- [Where Does Robustness Live? Neuron-Guided Adaptation for Retrieval-Augmented Language Models](https://mlsanity.com/paper/2604.02194/) (2026)
- [Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.11217/) (2026)
- [SCoNE: Selective Context-aware Neuron Editing for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.00689/) (2026)

## Abstract (arXiv)

The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective Preference Alignment for Retrieval-Augmented Generation (PA-RAG), a method for optimizing the generator of RAG systems to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at https://github.com/wujwyi/PA-RAG.