FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models
Not on the map. Why?
Abstract (arXiv)
To process contexts with unlimited length using Large Language Models (LLMs), recent studies explore hierarchically managing the long text. Only several text fragments are taken from the external memory and passed into the temporary working memory, i.e., LLM's context window. However, existing approaches isolatedly handle the text fragments without considering their structural connections, thereby suffering limited capability on texts with intensive inter-relations, e.g., coherent stories and code repositories. This work attempts to resolve this by exploiting the fragment-level relations in external memory. First, we formulate the fragment-level relations and present several instantiations for different text types. Next, we introduce a relation-aware fragment assessment criteria upon previous independent fragment assessment. Finally, we present the fragment-connected Hierarchical Memory based LLM. We validate the benefits of involving these relations on long story understanding, repository-level code generation, and long-term chatting.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 5 Jun 2024.
Builds on
29 of 40 references.
References · 29
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
17 more
- 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
- 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
- 2023In-Context Retrieval-Augmented Language Modelscited by 694
- 2017The NarrativeQA Reading Comprehension Challengecited by 590
- 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
- 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
- 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
- 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
- 2022Large Language Models with Controllable Working Memorycited by 169
- 2022ReACC: A Retrieval-Augmented Code Completion Frameworkcited by 117
- 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
- 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62
- 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
- 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
- 2023Memory Augmented Large Language Models are Computationally Universalcited by 45
Built on by
Citing papers per year: 2026: 1.
1 paper.