# MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs

arXiv: https://arxiv.org/abs/2506.15215 · HTML: https://mlsanity.com/paper/2506.15215/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences](https://mlsanity.com/paper/2306.07906/) (2023)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation](https://mlsanity.com/paper/2410.03608/) (2024)
- [OpenCQA: Open-ended Question Answering with Charts](https://mlsanity.com/paper/2210.06628/) (2022)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [Testing AI on language comprehension tasks reveals insensitivity to underlying meaning](https://mlsanity.com/paper/2302.12313/) (2023)
- [PEDANTS: Cheap but Effective and Interpretable Answer Equivalence](https://mlsanity.com/paper/2402.11161/) (2024)
- [Learning Personalized Alignment for Evaluating Open-ended Text Generation](https://mlsanity.com/paper/2310.03304/) (2023)
- [ANTIQUE: A Non-Factoid Question Answering Benchmark](https://mlsanity.com/paper/1905.08957/) (2019)
- [Atomic Fact Decomposition Helps Attributed Question Answering](https://mlsanity.com/paper/2410.16708/) (2024)
- [Accurate and Nuanced Open-QA Evaluation Through Textual Entailment](https://mlsanity.com/paper/2405.16702/) (2024)
- [LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMs](https://mlsanity.com/paper/2409.14744/) (2024)

## Built on by (most cited first, 1 of 1)

- [Skill-based Agentic Evaluation for Real-time Data Science Tasks](https://mlsanity.com/paper/2609.16487/) (2026)

## Abstract (arXiv)

Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose MinosEval, a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.