# Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering

arXiv: https://arxiv.org/abs/2311.06503 · HTML: https://mlsanity.com/paper/2311.06503/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 54 of its 86 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering](https://mlsanity.com/paper/2104.06378/) (2021)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [KGAT: Knowledge Graph Attention Network for Recommendation](https://mlsanity.com/paper/1905.07854/) (2019)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)

## Built on by (most cited first, 29 of 29)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.18676/) (2024)
- [NativE: Multi-modal Knowledge Graph Completion in the Wild](https://mlsanity.com/paper/2406.17605/) (2024)
- [Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning](https://mlsanity.com/paper/2405.16869/) (2024)
- [Large Knowledge Model: Perspectives and Challenges](https://mlsanity.com/paper/2312.02706/) (2023)
- [Large Language Models for Social Networks: Applications, Challenges, and Solutions](https://mlsanity.com/paper/2401.02575/) (2024)
- [KnowTuning: Knowledge-aware Fine-tuning for Large Language Models](https://mlsanity.com/paper/2402.11176/) (2024)
- [Turning Your Strength into Watermark: Watermarking Large Language Model via Knowledge Injection](https://mlsanity.com/paper/2311.09535/) (2023)
- [Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2402.15444/) (2024)
- [Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2404.00942/) (2024)
- [Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks](https://mlsanity.com/paper/2405.13085/) (2024)
- [OntoTune: Ontology-Driven Self-training for Aligning Large Language Models](https://mlsanity.com/paper/2502.05478/) (2025)
- [ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization](https://mlsanity.com/paper/2402.09320/) (2024)
- [PersLLM: A Personified Training Approach for Large Language Models](https://mlsanity.com/paper/2407.12393/) (2024)
- [DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMs](https://mlsanity.com/paper/2406.18535/) (2024)
- [Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning](https://mlsanity.com/paper/2507.19586/) (2025)
- [Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering](https://mlsanity.com/paper/2508.15213/) (2025)
- [The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systems](https://mlsanity.com/paper/2510.24831/) (2025)
- [Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking](https://mlsanity.com/paper/2501.00244/) (2024)
- [Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection](https://mlsanity.com/paper/2502.06148/) (2025)
- [A Dynamic Knowledge Update-Driven Model with Large Language Models for Fake News Detection](https://mlsanity.com/paper/2509.11687/) (2025)
- [Multimodal Representation Learning Conditioned on Semantic Relations](https://mlsanity.com/paper/2508.17497/) (2025)
- [SAG: Style-Aligned Article Generation via Model Collaboration](https://mlsanity.com/paper/2410.03137/) (2024)
- [DMA: Online RAG Alignment with Human Feedback](https://mlsanity.com/paper/2511.04880/) (2025)
- [Panning for Gold: Expanding Domain-Specific Knowledge Graphs with General Knowledge](https://mlsanity.com/paper/2601.10485/) (2026)
- [ReAD: Reinforcement-Guided Capability Distillation for Large Language Models](https://mlsanity.com/paper/2605.11290/) (2026)
- [Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation](https://mlsanity.com/paper/2606.28998/) (2026)
- [ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues](https://mlsanity.com/paper/2607.01242/) (2026)

## Abstract (arXiv)

Deploying large language models (LLMs) to real scenarios for domain-specific question answering (QA) is a key thrust for LLM applications, which poses numerous challenges, especially in ensuring that responses are both accommodating to user requirements and appropriately leveraging domain-specific knowledge bases. They are the two major difficulties for LLM application as vanilla fine-tuning falls short of addressing. Combining these requirements, we conceive of them as the requirement for the model's preference to be harmoniously aligned with humans'. Thus, we introduce Knowledgeable Preference AlignmenT (KnowPAT), which constructs two kinds of preference sets to tackle the two issues. Besides, we design a new alignment objective to align the LLM preference with different human preferences uniformly, aiming to optimize LLM performance in real-world, domain-specific QA settings. Adequate experiments and comprehensive comparisons with 15 baseline methods illustrate that our KnowPAT is a superior pipeline for real-scenario domain-specific QA with LLMs.