# Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?

arXiv: https://arxiv.org/abs/2402.11597 · HTML: https://mlsanity.com/paper/2402.11597/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 54 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 30 of 46)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)

## Built on by (most cited first, 30 of 30)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)
- [Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning](https://mlsanity.com/paper/2408.13457/) (2024)
- [Fine-tuning Large Language Models with Sequential Instructions](https://mlsanity.com/paper/2403.07794/) (2024)
- [The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2406.19999/) (2024)
- [Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English](https://mlsanity.com/paper/2503.04099/) (2025)
- [Advancing Single and Multi-task Text Classification through Large Language Model Fine-tuning](https://mlsanity.com/paper/2412.08587/) (2024)
- [LookAlike: Consistent Distractor Generation in Math MCQs](https://mlsanity.com/paper/2505.01903/) (2025)
- [Evaluating LLMs with Multiple Problems at once](https://mlsanity.com/paper/2406.10786/) (2024)
- [Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions](https://mlsanity.com/paper/2411.10163/) (2024)
- [Detecting Linguistic Indicators for Stereotype Assessment with Large Language Models](https://mlsanity.com/paper/2502.19160/) (2025)
- [Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation](https://mlsanity.com/paper/2505.09027/) (2025)
- [Multi-Step Reasoning in Korean and the Emergent Mirage](https://mlsanity.com/paper/2501.05712/) (2025)
- [ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure](https://mlsanity.com/paper/2602.01472/) (2026)
- [MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness](https://mlsanity.com/paper/2504.21773/) (2025)
- [ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities](https://mlsanity.com/paper/2506.02480/) (2025)
- [PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries](https://mlsanity.com/paper/2506.18728/) (2025)
- [CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics](https://mlsanity.com/paper/2508.20643/) (2025)
- [Reasoning Gets Harder for LLMs Inside A Dialogue](https://mlsanity.com/paper/2603.20133/) (2026)
- [Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length](https://mlsanity.com/paper/2603.22608/) (2026)
- [QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Models](https://mlsanity.com/paper/2512.08646/) (2025)
- [UsefulBench: Towards Decision-Useful Information as a Target for Information Retrieval](https://mlsanity.com/paper/2604.15827/) (2026)
- [Zero-Indexing Internet Search Augmented Generation for Large Language Models](https://mlsanity.com/paper/2411.19478/) (2024)
- [SPADE: A Large Language Model Framework for Soil Moisture Pattern Recognition and Anomaly Detection in Precision Agriculture](https://mlsanity.com/paper/2509.18123/) (2025)
- [Measuring the State of Open Science in Transportation Using Large Language Models](https://mlsanity.com/paper/2601.14429/) (2026)
- [A Dual-Task Paradigm to Investigate Sentence Comprehension Strategies in Language Models](https://mlsanity.com/paper/2604.26351/) (2026)
- [Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting](https://mlsanity.com/paper/2608.02681/) (2026)
- [In-Place Instruction Following in Diffusion Language Models](https://mlsanity.com/paper/2609.07160/) (2026)
- [Faithful and Complete LLM-Assisted Automated Construction of Hierarchical Hardware Knowledge Graphs](https://mlsanity.com/paper/2610.12762/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are typically prompted to follow a single instruction per inference call. In this work, we analyze whether LLMs also hold the capability to handle multiple instructions simultaneously, denoted as Multi-Task Inference. For this purpose, we introduce the MTI Bench(Multi-Task Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000 instances across 25 tasks. Each task in the MTI Bench involves 2 to 3 sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces the total inference time by 1.46 times in average since it does not require multiple inference calls. Interestingly, contrary to the expectation that LLMs would perform better when tasks are divided, we find that state-of-the-art LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved performance with Multi-Task Inference compared to Single-Task Inference on the MTI Bench. We release the MTI Bench dataset and our code at this link https://github.com/guijinSON/MTI-Bench.