mlsanityData to 14 Oct 2026

Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?

arXiv 2402.11597 ↗First posted 18 Feb 2024cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as Multi-Task Inference. For this purpose, we introduce the MTI Bench(Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI Bench involves 2 to 3 sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces the total inference time by 1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-of-the-art LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved performance with Multi-Task Inference compared to Single-Task Inference on the MTI Bench. We release the MTI Bench dataset and our code at this link https://github.com/guijinSON/MTI-Bench.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

46 of 54 references.

References · 46

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
34 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  13. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  14. 2019Abductive Commonsense Reasoningcited by 301
  15. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  16. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  17. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  18. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  19. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  20. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  21. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  22. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  23. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  24. 2019Counterfactual Story Reasoning and Generationcited by 88
  25. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 85
  26. 2019Reasoning Over Paragraph Effects in Situationscited by 79
  27. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 76
  28. 2018DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehensioncited by 71
  29. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  30. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  31. 2018MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledgecited by 30
  32. 2017Measuring the Similarity of Sentential Arguments in Dialogcited by 23
  33. 2023ThoughtSource: A central hub for large language model reasoning datacited by 23
  34. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 9.

30 papers.

Sort:

By area:

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  2. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  3. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  4. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 63
  5. 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
  6. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  7. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American Englishcited by 8
  8. 2024Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuningcited by 7
  9. 2025LookAlike: Consistent Distractor Generation in Math MCQscited by 7
  10. 2024Evaluating LLMs with Multiple Problems at oncecited by 5
  11. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 5
  12. 2025Detecting Linguistic Indicators for Stereotype Assessment with Large Language Modelscited by 5
18 more
  1. 2025Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generationcited by 5
  2. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  3. 2026ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressurecited by 4
  4. 2025MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awarenesscited by 3
  5. 2025ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilitiescited by 3
  6. 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queriescited by 3
  7. 2025CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensicscited by 3
  8. 2026Reasoning Gets Harder for LLMs Inside A Dialoguecited by 2
  9. 2026Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Lengthcited by 2
  10. 2025QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Modelscited by 1
  11. 2026UsefulBench: Towards Decision-Useful Information as a Target for Information Retrievalcited by 1
  12. 2024Zero-Indexing Internet Search Augmented Generation for Large Language Modelscited by 0
  13. 2025SPADE: A Large Language Model Framework for Soil Moisture Pattern Recognition and Anomaly Detection in Precision Agriculturecited by 0
  14. 2026Measuring the State of Open Science in Transportation Using Large Language Modelscited by 0
  15. 2026A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Modelscited by 0
  16. 2026Safety in Batches? Understanding and Mitigating Safety Failures in Batch Promptingcited by 0
  17. 2026In-Place Instruction Following in Diffusion Language Modelscited by 0
  18. 2026Faithful and Complete LLM-Assisted Automated Construction of Hierarchical Hardware Knowledge Graphscited by 0