mlsanityData to 14 Oct 2026

Rethinking Generative Large Language Model Evaluation for Semantic Comprehension

arXiv 2403.07872 ↗First posted 12 Mar 2024cs.CLcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Despite their sophisticated capabilities, large language models (LLMs) encounter a major hurdle in effective assessment. This paper first revisits the prevalent evaluation method-multiple choice question answering (MCQA), which allows for straightforward accuracy measurement. Through a comprehensive evaluation of 24 models across 11 benchmarks, we highlight several potential drawbacks of MCQA, for instance, the inconsistency between the MCQA evaluation and the generation of open-ended responses in practical scenarios. In response, we introduce an RWQ-Elo rating system, engaging 24 LLMs such as GPT-4, GPT-3.5, Google-Gemini-Pro and LLaMA-1/-2, in a two-player competitive format, with GPT-4 serving as the judge. Each LLM receives an Elo rating thereafter. This system is designed to mirror real-world usage, and for this purpose, we have compiled a new benchmark called ``Real-world questions'' (RWQ), comprising 20,772 authentic user inquiries. Additionally, we thoroughly analyze the characteristics of our system and compare it with prior leaderboards like AlpacaEval and MT-Bench. Our analysis reveals the stability of our RWQ-Elo system, the feasibility of registering new models, and its potential to reshape LLM leaderboards.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 71 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

38 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2016Layer Normalizationcited by 5,669
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021Program Synthesis with Large Language Modelscited by 3,637
25 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2018Hierarchical Neural Story Generationcited by 1,193
  14. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  15. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  17. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  18. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  19. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  20. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  21. 2023The Falcon Series of Open Language Modelscited by 535
  22. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  23. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  24. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 124
  25. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  2. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 36
  3. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  4. 2024LOGIC-LM++: Multi-Step Refinement for Symbolic Formulationscited by 14
  5. 2024Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Frameworkcited by 9
  6. 2024Awes, Laws, and Flaws From Today's LLM Researchcited by 5
  7. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  8. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  9. 2025The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarkscited by 3
  10. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1
  11. 2026Auditing MCQA Benchmarks through Probability Landscapescited by 1
  12. 2026StarDrinks: An English and Korean Test Set for SLU Evaluation in a Drink Ordering Scenariocited by 0
1 more
  1. 2026Likelihood Ranking doesn't Scale Like Prompting in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
  2. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
  3. 2024MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  5. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Models
  6. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  7. 2025Rethinking Data: Towards Better Performing Domain-Specific Small Language Models
  8. 2023On the Calibration of Multilingual Question Answering LLMs