mlsanityData to 14 Oct 2026

LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

arXiv 2602.23603 ↗First posted 27 Feb 2026cs.CL cs.AI cs.IRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Long-form question answering (LFQA) demands nuanced evaluation of multi-sentence explanatory responses, yet existing metrics often fail to reflect human judgment. We present LFQA-HP-1M, a large-scale dataset comprising 1.3M human pairwise preference annotations for LFQA. We propose nine rubrics for answer quality evaluation, and show that simple linear models based on these features perform comparably to state-of-the-art LLM evaluators. We further examine transitivity consistency, positional bias, and verbosity biases in LLM evaluators and demonstrate their vulnerability to adversarial perturbations. Overall, this work provides one of the largest public LFQA preference datasets and a rubric-driven framework for transparent and reliable evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 44 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

25 of 44 references.

References · 25

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  7. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  8. 2019ELI5: Long Form Question Answeringcited by 575
  9. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  11. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
13 more
  1. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  2. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  3. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  4. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  5. 2018Interpretation of Natural Language Rules in Conversational Machine Readingcited by 75
  6. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  7. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  8. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  9. 2021Token-Modification Adversarial Attacks for Natural Language Processing: A Surveycited by 12
  10. 2024SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysiscited by 8
  11. 2023Axiomatic Preference Modeling for Longform Question Answeringcited by 6
  12. 2024Localizing and Mitigating Errors in Long-form Question Answeringcited by 4
  13. 2025An Empirical Study of Evaluating Long-form Question Answeringcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
  2. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
  3. 2024DebateQA: Evaluating Question Answering on Debatable Knowledge
  4. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering
  5. 2023LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
  6. 2025Preference-based Learning with Retrieval Augmented Generation for Conversational Question Answering
  7. 2024Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter
  8. 2023ToolQA: A Dataset for LLM Question Answering with External Tools