mlsanityData to 7 Oct 2026

Evaluating LLMs with Multiple Problems at once

arXiv 2406.10786 ↗First posted 16 Jun 2024cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once, a paradigm we call multi-problem evaluation (MPE). Unlike conventional single-problem evaluation, where a prompt presents a single problem and expects one specific answer, MPE places multiple problems together in a single prompt and assesses how well an LLM answers all these problems in a single output. Leveraging 6 classification and 12 reasoning benchmarks that already exist, we introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts. We experiment with a total of 13 LLMs from 5 model families on ZeMPE to present a comprehensive and systematic MPE. Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short. In addition, we perform in-depth further analyses and explore model-level factors that may enable multiple problem handling capabilities in LLMs. We release our corpus and code to facilitate future research.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

32 of 55 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
20 more
  1. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2018Neural Network Acceptability Judgmentscited by 952
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  5. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  9. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  10. 2016Solving General Arithmetic Word Problemscited by 353
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  12. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  13. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  14. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  15. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  16. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  17. 2023Large Language Models as Analogical Reasonerscited by 107
  18. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  19. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  20. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  2. 2024SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characterscited by 6
  3. 2026ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMscited by 5
  4. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
  5. 2025RWESummary: A Framework and Test for Choosing Large Language Models to Summarize Real-World Evidence (RWE) Studiescited by 0