# What is the best model? Application-driven Evaluation for Large Language Models

arXiv: https://arxiv.org/abs/2406.10307 · HTML: https://mlsanity.com/paper/2406.10307/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 17 of 17)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [ARB: Advanced Reasoning Benchmark for Large Language Models](https://mlsanity.com/paper/2307.13692/) (2023)
- [Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning](https://mlsanity.com/paper/2306.02408/) (2023)
- [M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models](https://mlsanity.com/paper/2305.10263/) (2023)
- [Measuring Massive Multitask Chinese Understanding](https://mlsanity.com/paper/2304.12986/) (2023)
- [OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety](https://mlsanity.com/paper/2403.12316/) (2024)
- [Towards Effective Ancient Chinese Translation: Dataset, Model, and Evaluation](https://mlsanity.com/paper/2308.00240/) (2023)

## Built on by (most cited first, 4 of 4)

- [Methodology of Adapting Large English Language Models for Specific Cultural Contexts](https://mlsanity.com/paper/2406.18192/) (2024)
- [Quantifying the Capability Boundary of DeepSeek Models: An Application-Driven Performance Analysis](https://mlsanity.com/paper/2502.11164/) (2025)
- [TeleEval-OS: Performance evaluations of large language models for operations scheduling](https://mlsanity.com/paper/2506.11017/) (2025)
- [MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance](https://mlsanity.com/paper/2509.09730/) (2025)

## Abstract (arXiv)

General large language models enhanced with supervised fine-tuning and reinforcement learning from human feedback are increasingly popular in academia and industry as they generalize foundation models to various practical tasks in a prompt manner. To assist users in selecting the best model in practical application scenarios, i.e., choosing the model that meets the application requirements while minimizing cost, we introduce A-Eval, an application-driven LLMs evaluation benchmark for general large language models. First, we categorize evaluation tasks into five main categories and 27 sub-categories from a practical application perspective. Next, we construct a dataset comprising 678 question-and-answer pairs through a process of collecting, annotating, and reviewing. Then, we design an objective and effective evaluation method and evaluate a series of LLMs of different scales on A-Eval. Finally, we reveal interesting laws regarding model scale and task difficulty level and propose a feasible method for selecting the best model. Through A-Eval, we provide clear empirical and engineer guidance for selecting the best model, reducing barriers to selecting and using LLMs and promoting their application and development. Our benchmark is publicly available at https://github.com/UnicomAI/DataSet/tree/main/TestData/GeneralAbility.