# Revisit to Segment: Working Memory Distillation for Reasoning Segmentation

arXiv: https://arxiv.org/abs/2609.34863 · HTML: https://mlsanity.com/paper/2609.34863/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models](https://mlsanity.com/paper/2601.18734/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.10674/) (2026)
- [SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.14777/) (2026)
- [Multi-Rollout On-Policy Distillation via Peer Successes and Failures](https://mlsanity.com/paper/2605.12652/) (2026)
- [Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation](https://mlsanity.com/paper/2605.12741/) (2026)
- [Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search](https://mlsanity.com/paper/2602.04454/) (2026)
- [SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation](https://mlsanity.com/paper/2605.22658/) (2026)
- [DR2Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models](https://mlsanity.com/paper/2601.09981/) (2026)
- [Discriminative Perception via Anchored Description for Reasoning Segmentation](https://mlsanity.com/paper/2603.04002/) (2026)
- [StAR: Segment Anything Reasoner](https://mlsanity.com/paper/2603.14382/) (2026)
- [Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token](https://mlsanity.com/paper/2603.19026/) (2026)
- [From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding](https://mlsanity.com/paper/2605.15951/) (2026)
- [Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning](https://mlsanity.com/paper/2606.09303/) (2026)
- [Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification](https://mlsanity.com/paper/2605.17531/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have approached image segmentation by reasoning about visual content and predicting target locations. Their generated responses contain reasoning traces and localization proposals that can serve as working memory when revisiting the same image and query. Our exploration reveals that MLLMs benefit from using this self-generated working memory as context, leading to enhanced reasoning segmentation. Motivated by this finding, we seek to strengthen the backbone model's reasoning segmentation capabilities by distilling the guidance gained from revisiting prior attempts, enabling it to benefit with or without working memory at inference time. To this end, we propose Reasoning Segmenter with Working Memory (SWiM), a working-memory distillation framework for reasoning segmentation. Specifically, SWiM selects rollouts based on segmentation quality to construct working memory and uses the memory-conditioned model as a teacher. The teacher provides token-level distributional supervision along student-generated trajectories, while the student receives only the original image and query. Joint optimization of on-policy self-distillation and outcome-based reinforcement learning combines working-memory guidance with direct feedback on segmentation quality. Extensive experiments on reasoning segmentation benchmarks demonstrate that SWiM achieves state-of-the-art performance, validating the effectiveness of working-memory distillation.