Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models
Not on the map. Why?
Abstract (arXiv)
Long-context language models (LCLMs) have exhibited impressive capabilities in long-context understanding tasks. Among these, long-context referencing -- a crucial task that requires LCLMs to attribute items of interest to specific parts of long-context data -- remains underexplored. To bridge this gap, this paper proposes Referencing Evaluation for Long-context Language Models (Ref-Long), a novel benchmark designed to assess the long-context referencing capability of LCLMs. Specifically, Ref-Long requires LCLMs to identify the indexes of documents that reference a specific key, emphasizing contextual relationships between the key and the documents over simple retrieval. Based on the task design, we construct three subsets ranging from synthetic to realistic scenarios to form the Ref-Long benchmark. Experimental results of 13 LCLMs reveal significant shortcomings in long-context referencing, even among advanced models like GPT-4o. To further investigate these challenges, we conduct comprehensive analyses, including human evaluations, task format adjustments, fine-tuning experiments, and error analyses, leading to several key insights. Our data and code can be found in https://github. com/wujunjie1998/Ref-Long.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 13 Jul 2025.
Builds on
24 of 34 references.
References · 24
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
- 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
- 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
- 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
- 2024How to Train Long-Context Language Models (Effectively)cited by 146
- 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
- 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
12 more
- 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
- 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
- 2024LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QAcited by 81
- 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
- 2021Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversationscited by 20
- 2023Personality Understanding of Fictional Characters during Book Readingcited by 20
- 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
- 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
- 2024Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questionscited by 11
- 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
- 2024L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?cited by 5
- 2025Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Taskcited by 4
Built on by
Citing papers per year: 2025: 2.
2 papers.