# LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

arXiv: https://arxiv.org/abs/2502.14644 · HTML: https://mlsanity.com/paper/2502.14644/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 56 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [MovieQA: Understanding Stories in Movies through Question-Answering](https://mlsanity.com/paper/1512.02902/) (2015)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [Test-Time Training with Masked Autoencoders](https://mlsanity.com/paper/2209.07522/) (2022)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://mlsanity.com/paper/2311.04939/) (2023)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [FLatten Transformer: Vision Transformer using Focused Linear Attention](https://mlsanity.com/paper/2308.00442/) (2023)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)
- [Retrieval meets Long Context Large Language Models](https://mlsanity.com/paper/2310.03025/) (2023)
- [LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs](https://mlsanity.com/paper/2406.15319/) (2024)

## Built on by (most cited first, 6 of 6)

- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [ContextNav: Towards Agentic Multimodal In-Context Learning](https://mlsanity.com/paper/2510.04560/) (2025)
- [ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time](https://mlsanity.com/paper/2507.06313/) (2025)
- [S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference](https://mlsanity.com/paper/2601.17702/) (2026)
- [RAG-Enhanced Large Language Models for Dynamic Content Expiration Prediction in Web Search](https://mlsanity.com/paper/2605.13052/) (2026)
- [CreAgentive: An Agent Workflow Driven Multi-Category Creative Generation Engine](https://mlsanity.com/paper/2509.26461/) (2025)

## Abstract (arXiv)

Long-context understanding remains challenging for LLMs due to limited context windows. This paper introduces Long Input Fine-Tuning (LIFT), a framework that improves the long-context performance of arbitrary short-context LLMs by dynamically adapting their parameters to each long input. Instead of endlessly extending context windows to fit longer inputs in context, LIFT stores and absorbs the input in parameters. By fine-tuning long inputs into parameters, LIFT enables short-context LLMs to answer questions even when required information is absent from the inference context, avoiding the quadratic input-length complexity of standard long-context models. Rather than simple continued pretraining on new long contexts, LIFT uses carefully designed LLM-generated synthetic tasks to enhance comprehension beyond memorization. To offset fine-tuning overhead, we design a highly optimized pipeline that reduces Time to First Token (TTFT) to under 10 seconds for 8k context. We further analyze LIFT's strengths and limitations, discuss large-scale deployment feasibility, and highlight future research directions. Implementation is open-sourced at https://github.com/MuLabPKU/LIFT.