mlsanityData to 14 Oct 2026

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

arXiv 2409.06679 ↗First posted 10 Sep 2024cs.CLcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 58 of its 73 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

58 of 73 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  11. 2022A Survey on In-context Learningcited by 721
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  13. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  14. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  15. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  16. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  17. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  18. 2023Adapting Language Models to Compress Contextscited by 356
  19. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  20. 2021Efficient Attentions for Long Document Summarizationcited by 345
  21. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  22. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  23. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  24. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  25. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  26. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  27. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  28. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  29. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  30. 2022A Length-Extrapolatable Transformercited by 127
  31. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  32. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 112
  33. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  34. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  35. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  36. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 65
  37. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  38. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 7.

13 papers.

Sort:

By area:

  1. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  2. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  3. 2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
  4. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  5. 2026AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completioncited by 7
  6. 2026End-to-End Context Compression at Scalecited by 4
  7. 2025C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Poolingcited by 3
  8. 2026DRAINCODE: Stealthy Energy Consumption Attacks on Retrieval-Augmented Code Generation via Context Poisoningcited by 3
  9. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
  10. 2025Understanding Parametric Knowledge Injection in Retrieval-Augmented Generationcited by 1
  11. 2026Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inferencecited by 1
  12. 2026UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inferencecited by 1
1 more
  1. 2026VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  2. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  4. 2024FocusLLM: Precise Understanding of Long Context by Dynamic Condensing
  5. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models
  6. 2024ACER: Automatic Language Model Context Extension via Retrieval
  7. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
  8. 2024Large Language Models Can Self-Improve in Long-context Reasoning