mlsanityData to 14 Oct 2026

Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing

arXiv 2505.08651 ↗First posted 13 May 2025cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We present MegaBeam-Mistral-7B, a language model that supports 512K-token context length. Our work addresses practical limitations in long-context training, supporting real-world tasks such as compliance monitoring and verification. Evaluated on three long-context benchmarks, our 7B-parameter model demonstrates superior in-context learning performance on HELMET and robust retrieval and tracing capability on RULER. It is currently the only open model to achieve competitive long-range reasoning on BABILong at 512K context length without RAG or targeted fine-tuning. Released as fully open source under the Apache 2.0 license, the model has been downloaded over 100,000 times on Hugging Face. Model available at: https://huggingface.co/aws-prototyping/MegaBeam-Mistral-7B-512k

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 19 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

13 of 19 references.

References · 13

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  5. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  6. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  8. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  9. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  10. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  11. 2024Base of RoPE Bounds Context Lengthcited by 52
  12. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 25
1 more
  1. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025SeqPE: Transformer with Sequential Position Encodingcited by 2
  2. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  2. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models
  3. 2026LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
  4. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
  6. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  7. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  8. 2026Trained Agentic Context Management