mlsanityData to 7 Oct 2026

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

arXiv 2307.10928 ↗First posted 20 Jul 2023cs.CL cs.AIcited by 198

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations. We publicly release the evaluation data and code implementation at https://github.com/kaistAI/FLASK.

Key facts. Cited by 198 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 113 of its 161 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

113 of 161 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Aligning Large Language Models through Synthetic Feedback23 May
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 108

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  13. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  14. 2018Hierarchical Neural Story Generationcited by 1,182
  15. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  16. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  17. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  18. 2023StarCoder: may the source be with you!cited by 790
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  21. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  22. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  23. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  24. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  25. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  26. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  27. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  28. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  29. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  30. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  31. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  32. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  33. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  34. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  35. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  36. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  37. 2019Abductive Commonsense Reasoningcited by 300
  38. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 60; 2025: 52; 2026: 75.

198 papers.

Sort:

By area:

  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  5. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  6. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  7. 2023Large Language Model Alignment: A Surveycited by 241
  8. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  9. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  10. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  11. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  12. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
38 more
  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  2. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  3. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  4. 2023Human Feedback is not Gold Standardcited by 82
  5. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  6. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  7. 2024Self-Taught Evaluatorscited by 70
  8. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 65
  9. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  10. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  11. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 46
  12. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  13. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  14. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  15. 2025KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systemscited by 36
  16. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  17. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  18. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  19. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  20. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  21. 2024Direct Judgement Preference Optimizationcited by 33
  22. 2024INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Modelscited by 32
  23. 2024Finding Blind Spots in Evaluator LLMs with Interpretable Checklistscited by 32
  24. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  25. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  26. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  27. 2023HARE: Explainable Hate Speech Detection with Step-by-Step Reasoningcited by 30
  28. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  29. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  30. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  31. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  32. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  33. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  34. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 21
  35. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  36. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 20
  37. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  38. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19

50 most cited of 198 All 198 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
  3. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
  4. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  5. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  6. 2024What is the best model? Application-driven Evaluation for Large Language Models
  7. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  8. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback