AcademicEval: Live Long-Context LLM Benchmark

Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose textscAcademicEval, a live benchmark for evaluating LLMs over long-context generation tasks. textscAcademicEval adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, textiti.e., textscTitle, textscAbstract, textscIntroduction, and textscRelated Work, which cover a wide range of abstraction levels and require no manual labeling. Moreover, textscAcademicEval integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, textscAcademicEval features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on textscAcademicEval, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at https://github.com/ulab-uiuc/AcademicEval