mlsanityData to 7 Oct 2026

Focused Transformer: Contrastive Training for Context Scaling

arXiv 2307.03170 ↗First posted 6 Jul 2023cs.CL cs.AI cs.LGcited by 133

Not on the map. Why?

Abstract (arXiv)

Large language models have an exceptional capability to incorporate new information in a contextual manner. However, the full potential of such an approach is often restrained due to a limitation in the effective context length. One solution to this issue is to endow an attention layer with access to an external memory, which comprises of (key, value) pairs. Yet, as the number of documents increases, the proportion of relevant keys to irrelevant ones decreases, leading the model to focus more on the irrelevant keys. We identify a significant challenge, dubbed the distraction issue, where keys linked to different semantic values might overlap, making them hard to distinguish. To tackle this problem, we introduce the Focused Transformer (FoT), a technique that employs a training process inspired by contrastive learning. This novel approach enhances the structure of the (key, value) space, enabling an extension of the context length. Our method allows for fine-tuning pre-existing, large-scale models to lengthen their effective context. This is demonstrated by our fine-tuning of 3B and 7B OpenLLaMA checkpoints. The resulting models, which we name LongLLaMA, exhibit advancements in tasks requiring a long context. We further illustrate that our LongLLaMA models adeptly manage a 256 k context length for passkey retrieval.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

39 of 56 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
27 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2019Root Mean Square Layer Normalizationcited by 1,425
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  12. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  13. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  14. 2020Query-Key Normalization for Transformerscited by 266
  15. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  16. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  17. 2022Memorizing Transformerscited by 219
  18. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  19. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  20. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  21. 2022Training Language Models with Memory Augmentationcited by 112
  22. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  23. 2022Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Proverscited by 97
  24. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  25. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  26. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  27. 2023Magnushammer: A Transformer-Based Approach to Premise Selectioncited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 69; 2025: 28; 2026: 11.

133 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  3. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  4. 2024Large Language Models: A Surveycited by 506
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  9. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  10. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  11. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  12. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
38 more
  1. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  2. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  3. 2023Efficient Large Language Models: A Surveycited by 183
  4. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  5. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  6. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  7. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  8. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  9. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  10. 2024Make Your LLM Fully Utilize the Contextcited by 118
  11. 2023Retrieval meets Long Context Large Language Modelscited by 114
  12. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  13. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  14. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  15. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
  16. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  17. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  18. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  19. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  20. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 74
  21. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  22. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  23. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  24. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  25. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  26. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  27. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  28. 2024Base of RoPE Bounds Context Lengthcited by 49
  29. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  30. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 45
  31. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  32. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  33. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  34. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  35. 2024Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysiscited by 33
  36. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  37. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  38. 2024LongGenBench: Long-context Generation Benchmarkcited by 27

50 most cited of 133 All 133 (JSON)