mlsanityData to 14 Oct 2026

Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation

arXiv 2410.18001 ↗First posted 23 Oct 2024cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Foundation models (FMs) have achieved significant success across various tasks, leading to research on benchmarks for reasoning abilities. However, there is a lack of studies on FMs performance in exceptional scenarios, which we define as out-of-distribution (OOD) reasoning tasks. This paper is the first to address these cases, developing a novel dataset for evaluation of FMs across multiple modalities, including graphic novels, calligraphy, news articles, and lyrics. It includes tasks for instance classification, character recognition, token prediction, and text generation. The paper also proposes prompt engineering techniques like Chain-of-Thought (CoT) and CoT+Few-Shot to enhance performance. Validation of FMs using various methods revealed improvements. The code repository is accessible at: https://github.com/MLAI-Yonsei/ExceptionalBenchmark

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 20 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

15 of 20 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  12. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
3 more
  1. 2016Visual Storytellingcited by 176
  2. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  3. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Eureka: Evaluating and Understanding Large Foundation Models
  2. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
  3. 2026Unified Deployment-Aware Evaluation of Open Reasoning Language Models
  4. 2026Reliable and Responsible Foundation Models: A Comprehensive Survey
  5. 2025Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It
  6. 2025ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning
  7. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
  8. 2024Surprisingly Fragile: Assessing and Addressing Prompt Instability in Multimodal Foundation Models