# Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking

arXiv: https://arxiv.org/abs/2602.13980 · HTML: https://mlsanity.com/paper/2602.13980/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Learning to Compress Prompt in Natural Language Formats](https://mlsanity.com/paper/2402.18700/) (2024)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [PISCO: Pretty Simple Compression for Retrieval-Augmented Generation](https://mlsanity.com/paper/2501.16075/) (2025)
- [Synaptic Theory of Chunking in Working Memory](https://mlsanity.com/paper/2408.07637/) (2024)

## Built on by (most cited first, 1 of 1)

- [Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG](https://mlsanity.com/paper/2609.05152/) (2026)

## Abstract (arXiv)

Providing extensive context via prompting is vital for leveraging the capabilities of Large Language Models (LLMs). However, lengthy contexts significantly increase inference latency, as the computational cost of self-attention grows quadratically with sequence length. To mitigate this issue, context compression-particularly soft prompt compressio-has emerged as a widely studied solution, which converts long contexts into shorter memory embeddings via a trained compressor. Existing methods typically compress the entire context indiscriminately into a set of memory tokens, requiring the compressor to capture global dependencies and necessitating extensive pre-training data to learn effective patterns. Inspired by the chunking mechanism in human working memory and empirical observations of the spatial specialization of memory embeddings relative to original tokens, we propose Parallelized Iterative Compression (PIC). By simply modifying the Transformer's attention mask, PIC explicitly restricts the receptive field of memory tokens to sequential local chunks, thereby lowering the difficulty of compressor training. Experiments across multiple downstream tasks demonstrate that PIC consistently outperforms competitive baselines, with superiority being particularly pronounced in high compression scenarios (e.g., achieving relative improvements of 29.8% in F1 score and 40.7% in EM score on QA tasks at the 64× compression ratio). Furthermore, PIC significantly expedites the training process. Specifically, when training the 16× compressor, it surpasses the peak performance of the competitive baseline while effectively reducing the training time by approximately 40%.