mlsanityData to 7 Oct 2026

Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?

arXiv 2405.06105 ↗First posted 9 May 2024cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Recent studies have shown that Large Language Models (LLMs) have the potential to process extremely long text. Many works only evaluate LLMs' long-text processing ability on the language modeling task, with perplexity (PPL) as the evaluation metric. However, in our study, we find that there is no correlation between PPL and LLMs' long-text understanding ability. Besides, PPL may only reflect the model's ability to model local information instead of catching long-range dependency. Therefore, only using PPL to prove the model could process long text is inappropriate. The local focus feature of PPL could also explain some existing phenomena, such as the great extrapolation ability of the position method ALiBi. When evaluating a model's ability in long text, we might pay more attention to PPL's limitation and avoid overly relying on it.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

7 of 9 references.

References · 7

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  6. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  7. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 21; 2026: 9.

45 papers.

Sort:

By area:

  1. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  4. 2024Base of RoPE Bounds Context Lengthcited by 49
  5. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 46
  6. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  7. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  8. 2025Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learningcited by 25
  9. 2025GraphRAG under Firecited by 23
  10. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  11. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practicescited by 22
  12. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
33 more
  1. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  2. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  3. 2025Token Weighting for Long-Range Language Modelingcited by 14
  4. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 13
  5. 2024Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Modelscited by 13
  6. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  7. 2024GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMscited by 10
  8. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  9. 2024Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Modelscited by 9
  10. 2024MELODI: Exploring Memory Compression for Long Contextscited by 9
  11. 2026Perplexity Cannot Always Tell Right from Wrongcited by 8
  12. 2025Hierarchical Tree Search-based User Lifelong Behavior Modeling on Large Language Modelcited by 6
  13. 2025Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Modelcited by 4
  14. 2024Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Modelscited by 3
  15. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  16. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesiscited by 3
  17. 2025Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolationcited by 3
  18. 2025AutoSpec: An Agentic Framework for Automatically Drafting Patent Specificationcited by 3
  19. 2024TALEC: Teach Your LLM to Evaluate in Specific Domain with In-house Criteria by Criteria Division and Zero-shot Plus Few-shotcited by 2
  20. 2025Controllably Efficient Language Modelscited by 2
  21. 2026EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuningcited by 2
  22. 2024Enhancing LLM Evaluations: The Garbling Trickcited by 1
  23. 2025iServe: An Intent-based Serving System for LLMscited by 1
  24. 2026A Theoretical Framework for Statistical Evaluability of Generative Modelscited by 1
  25. 2026Scale-Sensitive Shattering: Learnability and Evaluability at Optimal Scalecited by 1
  26. 2025Estimating the Effects of Sample Training Orders for Large Language Models without Retrainingcited by 0
  27. 2025A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancementcited by 0
  28. 2025COBRA: Catastrophic Bit-flip Reliability Analysis of State-Space Modelscited by 0
  29. 2026Meta-probabilistic Modelingcited by 0
  30. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  31. 2026CCS: Clinical Consensus Selection for Radiology Report Generationcited by 0
  32. 2026Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debatecited by 0
  33. 2026Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Promptingcited by 0