mlsanityData to 14 Oct 2026

INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

arXiv 2306.04757 ↗First posted 7 Jun 2023cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. Despite their impressive capabilities, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and the absence of holistic evaluation studies. To address these challenges, we present INSTRUCTEVAL, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is the most crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment. We are encouraged by the rapid development of models by the open-source community, but we also highlight the need for rigorous evaluation to support claims made about these models. Through INSTRUCTEVAL, we aim to foster a deeper understanding of instruction-tuned models and advancements in their capabilities. INSTRUCTEVAL is publicly available at https://github.com/declare-lab/instruct-eval.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

32 of 55 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
20 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  13. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  15. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  16. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  17. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  18. 2022UL2: Unifying Language Learning Paradigmscited by 295
  19. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  20. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 22; 2025: 14; 2026: 1.

62 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  2. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  6. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  7. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  8. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  9. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  10. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  11. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 90
  12. 2023Effective Prompt Extraction from Language Modelscited by 85
38 more
  1. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  2. 2023Efficient Benchmarking of Language Modelscited by 71
  3. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  4. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  5. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  6. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  7. 2023Dataset Quantizationcited by 48
  8. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  9. 2023Instruction-tuning Aligns LLMs to the Human Braincited by 40
  10. 2024Best Practices for Text Annotation with Large Language Modelscited by 36
  11. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  12. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  13. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  14. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  15. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 25
  16. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  17. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 22
  18. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 20
  19. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 19
  20. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
  21. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18
  22. 2024Machine Translation Meta Evaluation through Translation Accuracy Challenge Setscited by 18
  23. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  24. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyondcited by 14
  25. 2024FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?cited by 12
  26. 2024Look Within, Why LLMs Hallucinate: A Causal Perspectivecited by 11
  27. 2023The Poison of Alignmentcited by 10
  28. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  29. 2025Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankingscited by 8
  30. 2023ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizerscited by 6
  31. 2024LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuningcited by 6
  32. 2024Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approachcited by 5
  33. 2025DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?cited by 5
  34. 2025Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generationcited by 5
  35. 2024Harnessing the Power of Semi-Structured Knowledge and LLMs with Triplet-Based Prefiltering for Question Answeringcited by 4
  36. 2024Beyond Metrics: A Critical Analysis of the Variability in Large Language Model Evaluation Frameworkscited by 3
  37. 2024TabVer: Tabular Fact Verification with Natural Logiccited by 3
  38. 2025Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledgecited by 3

50 most cited of 62 All 62 (JSON)