OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions
Not on the map. Why?
Abstract (arXiv)
The rapid advancements in Large Language Models (LLMs) have significantly expanded their applications, ranging from multilingual support to domain-specific tasks and multimodal integration. In this paper, we present OmniEvalKit, a novel benchmarking toolbox designed to evaluate LLMs and their omni-extensions across multilingual, multidomain, and multimodal capabilities. Unlike existing benchmarks that often focus on a single aspect, OmniEvalKit provides a modular, lightweight, and automated evaluation system. It is structured with a modular architecture comprising a Static Builder and Dynamic Data Flow, promoting the seamless integration of new models and datasets. OmniEvalKit supports over 100 LLMs and 50 evaluation datasets, covering comprehensive evaluations across thousands of model-dataset combinations. OmniEvalKit is dedicated to creating an ultra-lightweight and fast-deployable evaluation framework, making downstream applications more convenient and versatile for the AI community.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 73 references that are in the map. First posted on arXiv 9 Dec 2024.
Builds on
32 of 73 references.
References · 32
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Qwen Technical Reportcited by 3,344
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024Qwen2 Technical Reportcited by 2,120
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
20 more
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2024Mixtral of Expertscited by 1,823
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
- 2022A Survey on In-context Learningcited by 721
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
- 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
- 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 162
- 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
- 2023EQ-Bench: An Emotional Intelligence Benchmark for Large Language Modelscited by 80
- 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
- 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
- 2023Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUEcited by 10
Built on by
Not cited yet.