# LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

arXiv: https://arxiv.org/abs/2507.04723 · HTML: https://mlsanity.com/paper/2507.04723/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 30 of 33)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration](https://mlsanity.com/paper/2410.02367/) (2024)
- [LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs](https://mlsanity.com/paper/2408.07055/) (2024)
- [XAttention: Block Sparse Attention with Antidiagonal Scoring](https://mlsanity.com/paper/2503.16428/) (2025)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference](https://mlsanity.com/paper/2502.20766/) (2025)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks](https://mlsanity.com/paper/2407.08454/) (2024)
- [LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K](https://mlsanity.com/paper/2402.05136/) (2024)
- [Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference](https://mlsanity.com/paper/2402.09398/) (2024)
- [CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences](https://mlsanity.com/paper/2503.12491/) (2025)
- [Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers](https://mlsanity.com/paper/2406.16747/) (2024)
- [LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion](https://mlsanity.com/paper/2501.15089/) (2025)
- [UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs](https://mlsanity.com/paper/2404.07584/) (2024)
- [Accurate KV Cache Quantization with Outlier Tokens Tracing](https://mlsanity.com/paper/2505.10938/) (2025)

## Built on by (most cited first, 5 of 5)

- [Revisiting Long-context Modeling from Context Denoising Perspective](https://mlsanity.com/paper/2510.05862/) (2025)
- [LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling](https://mlsanity.com/paper/2510.06915/) (2025)
- [Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers](https://mlsanity.com/paper/2601.17367/) (2026)
- [MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models](https://mlsanity.com/paper/2601.11969/) (2026)
- [Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference](https://mlsanity.com/paper/2604.07394/) (2026)

## Abstract (arXiv)

Long-context processing has become a fundamental capability for large language models~(LLMs). To assess model's long-context performance, numerous long-context evaluation benchmarks have been proposed. However, variations in evaluation settings across these benchmarks lead to inconsistent results, making it difficult to draw reliable comparisons. Besides, the high computational cost of long-context evaluation poses a significant barrier for the community to conduct comprehensive assessments of long-context models. In this paper, we propose LOOM-Scope, a comprehensive and efficient framework for long-context evaluation. LOOM-Scope standardizes evaluation settings across diverse benchmarks, supports deployment of efficient long-context inference acceleration methods, and introduces a holistic yet lightweight benchmark suite to evaluate models comprehensively. Homepage: https://loomscope.github.io