# Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length

arXiv: https://arxiv.org/abs/2603.22608 · HTML: https://mlsanity.com/paper/2603.22608/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 29 of 29)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing](https://mlsanity.com/paper/2410.12189/) (2024)
- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [Why Do Large Language Models (LLMs) Struggle to Count Letters?](https://mlsanity.com/paper/2412.18626/) (2024)
- [Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?](https://mlsanity.com/paper/2402.11597/) (2024)
- [How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://mlsanity.com/paper/2505.18761/) (2025)
- [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025)
- [MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents](https://mlsanity.com/paper/2508.11133/) (2025)
- [LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark](https://mlsanity.com/paper/2601.02872/) (2026)
- [More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG](https://mlsanity.com/paper/2503.04388/) (2025)
- [The WiLI benchmark dataset for written language identification](https://mlsanity.com/paper/1801.07779/) (2018)
- [Too Long, Didn't Model: Decomposing LLM Long-Context Understanding With Novels](https://mlsanity.com/paper/2505.14925/) (2025)
- [A Controllable Examination for Long-Context Language Models](https://mlsanity.com/paper/2506.02921/) (2025)

## Built on by (most cited first, 2 of 2)

- [Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media](https://mlsanity.com/paper/2605.21338/) (2026)
- [Beyond Retrieval: Evidence Access and Event Aggregation in Agent Memory](https://mlsanity.com/paper/2610.19474/) (2026)

## Abstract (arXiv)

Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform a comprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (approximately 20-100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.