Data to 7 Oct 2026

Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

arXiv 2409.04701 ↗First posted 7 Sep 2024cs.CL cs.IRcited by 70

Not on the map. Why?

Cited by 31 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 7 Sep 2024.

Builds on

16 of 24 references.

References · 16

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255

4 more

2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
2024Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retrievercited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 24; 2026: 44.

70 papers.

Sort:

By area:

2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
2025jina-reranker-v3: Last but Not Late Interaction for Listwise Document Rerankingcited by 23
2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
2024EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generationcited by 17
2025Should We Still Pretrain Encoders with Masked Language Modeling?cited by 15
2025Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generationcited by 12
2025Towards Reliable Retrieval in RAG Systems for Large Legal Datasetscited by 12
2025VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?cited by 11
2025MultiFinRAG: An Optimized Multimodal Retrieval-Augmented Generation (RAG) Framework for Financial Question Answeringcited by 9
2025HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunkingcited by 9
2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9

38 more

2025HeteRAG: A Heterogeneous Retrieval-augmented Generation Framework with Decoupled Knowledge Representationscited by 6
2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
2024LLM as HPC Expert: Extending RAG Architecture for HPC Datacited by 5
2026Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agentscited by 5
2025FinDebate: Multi-Agent Collaborative Intelligence for Financial Analysiscited by 4
2025AI for Climate Finance: Agentic Retrieval and Multi-Step Reasoning for Early Warning System Investmentscited by 3
2025CoRet: Improved Retriever for Code Editingcited by 3
2026Contrastive Learning with Narrative Twins for Modeling Story Saliencecited by 2
2026Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Biascited by 2
2025SpeakEasy: Enhancing Text-to-Speech Interactions for Expressive Content Creationcited by 1
2025OmniLLP: Enhancing LLM-based Log Level Prediction with Context-Aware Retrievalcited by 1
2025Cross-Document Topic-Aligned Chunking for Retrieval-Augmented Generationcited by 1
2025SmartChunk Retrieval: Query-Aware Chunk Compression with Planning for Efficient Document RAGcited by 1
2026KohakuRAG: A simple RAG framework with hierarchical document indexingcited by 1
2026Graph-Aware Late Chunking for Retrieval-Augmented Generation in Biomedical Literaturecited by 1
2026Adaptive Chunking: Optimizing Chunking-Method Selection for RAGcited by 1
2026From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documentscited by 1
2026SPIRE: Structure-Preserving Interpretable Retrieval of Evidencecited by 1
2026Reducing Redundancy in Retrieval-Augmented Generation through Chunk Filteringcited by 1
2026Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitationscited by 1
2026Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Textscited by 1
2026Right Reset: Chunking by Prefix Removalcited by 1
2025GeAR: Generation Augmented Retrievalcited by 0
2025SLIDE: Sliding Localized Information for Document Extractioncited by 0
2025Embedding-Based Context-Aware Rerankercited by 0
2025Democratizing ML for Enterprise Security: A Self-Sustained Attack Detection Frameworkcited by 0
2025M'as contexto no es mejor. Paradoja de la diluci'on vectorial en RAG corporativoscited by 0
2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0
2026Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Methodcited by 0
2025Clinical Document Metadata Extraction: A Scoping Reviewcited by 0
2026MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreadingcited by 0
2026AttentionRetriever: Attention Layers are Secretly Long Document Retrieverscited by 0
2026The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separationcited by 0
2026Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignmentcited by 0
2026Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networkscited by 0
2026A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspectioncited by 0
2026TopoChunker: Topology-Aware Agentic Document Chunking Frameworkcited by 0

50 most cited of 70 All 70 (JSON)