mlsanityData to 7 Oct 2026

Improving Attributed Text Generation of Large Language Models via Preference Learning

arXiv 2403.18381 ↗First posted 27 Mar 2024cs.CL cs.AIcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modelling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labelling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 45 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

32 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
19 more
  1. 2018Experience Replay for Continual Learningcited by 1,074
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2019ELI5: Long Form Question Answeringcited by 567
  7. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  8. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  10. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  11. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  12. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  13. 2023A Survey of Large Language Models Attributioncited by 71
  14. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  15. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  16. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  17. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  18. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  19. 2023Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attributioncited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 11; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  2. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 33
  3. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  4. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  5. 2024Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metricscited by 17
  6. 2024Advancing Large Language Model Attribution through Self-Improvingcited by 14
  7. 2025Lessons from Training Grounded LLMs with Verifiable Rewardscited by 8
  8. 2025Attribution, Citation, and Quotation: A Survey of Evidence-based Text Generation with Large Language Modelscited by 7
  9. 2024Citekit: A Modular Toolkit for Large Language Model Citation Generationcited by 5
  10. 2025Transparentize the Internal and External Knowledge Utilization in LLMs with Trustworthy Citationcited by 4
  11. 2025Towards Transparent RAG: Fostering Evidence Traceability in LLM Generation via Reinforcement Learningcited by 4
  12. 2025Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Familycited by 3
11 more
  1. 2025MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answeringcited by 3
  2. 2025HyperEdit: Unlocking Instruction-based Text Editing in LLMs via Hypernetworkscited by 2
  3. 2026GenProve: Learning to Generate Text with Fine-Grained Provenancecited by 2
  4. 2026Are Finer Citations Always Better? Rethinking Granularity for Attributed Generationcited by 2
  5. 2026Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScorecited by 1
  6. 2026CAGE: Cognitive Attribution Graphs for Faithful Inline Citation Generation in Long-Form Question Answeringcited by 1
  7. 2025ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Modelscited by 0
  8. 2025Citation Failure: Definition, Analysis and Efficient Mitigationcited by 0
  9. 2025CLINB: A Climate Intelligence Benchmark for Foundational Modelscited by 0
  10. 2025Insight-A: Attribution-aware for Multimodal Misinformation Detectioncited by 0
  11. 2026MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Usecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Role of Citations in Preference Data
  2. 2024Evaluation of Attribution Bias in Generator-Aware Retrieval-Augmented Large Language Models
  3. 2024Optimizing Language Models for Human Preferences is a Causal Inference Problem
  4. 2026Aligning Large Language Model Behavior with Human Citation Preferences
  5. 2026How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation
  6. 2024Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
  7. 2023Predicting Text Preference Via Structured Comparative Reasoning
  8. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators