mlsanityData to 7 Oct 2026

LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation

arXiv 2511.06346 ↗First posted 9 Nov 2025cs.AI cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) perform well on standard reasoning and question-answering benchmarks, yet such evaluations often fail to capture their ability to handle long-tail, expertise-intensive knowledge in real-world professional scenarios. We introduce LPFQA, a long-tail knowledge benchmark derived from authentic professional forum discussions, covering 7 academic and industrial domains with 430 curated tasks grounded in practical expertise. LPFQA evaluates specialized reasoning, domain-specific terminology understanding, and contextual interpretation, and adopts a hierarchical difficulty structure to ensure semantic clarity and uniquely identifiable answers. Experiments on over multiple mainstream LLMs reveal substantial performance gaps, particularly on tasks requiring deep domain reasoning, exposing limitations overlooked by existing benchmarks. Overall, LPFQA provides an authentic and discriminative evaluation framework that complements prior benchmarks and informs future LLM development.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 34 of its 50 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

34 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate22 May

Other references · 32

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
20 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  2. 2025Humanity's Last Examcited by 467
  3. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  4. 2021Deep Long-Tailed Learning: A Surveycited by 416
  5. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  6. 2023Using an LLM to Help With Code Understandingcited by 222
  7. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  8. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  9. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  10. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  11. 2024LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluationcited by 46
  12. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  13. 2024Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Factscited by 14
  14. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
  15. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Modelscited by 12
  16. 2024On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Modelscited by 10
  17. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  18. 2022Benchmarking Long-tail Generalization with Likelihood Splitscited by 8
  19. 2024Automatic Question-Answer Generation for Long-Tail Knowledgecited by 8
  20. 2021Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  2. 2026The Appeal and Reality of Recycling LoRAs with Adaptive Mergingcited by 2
  3. 2026DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Modelscited by 0
  4. 2026When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
  2. 2026Beyond Questions: Evaluating LLM's Knowledge Expression
  3. 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
  4. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
  5. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Models
  6. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
  7. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  8. 2025DeepQuestion: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance