mlsanityData to 7 Oct 2026

Text4Seg: Reimagining Image Segmentation as Text Generation

arXiv 2410.09855 ↗First posted 13 Oct 2024cs.CVcited by 59

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown exceptional capabilities in vision-language tasks; however, effectively integrating image segmentation into these models remains a significant challenge. In this paper, we introduce Text4Seg, a novel text-as-mask paradigm that casts image segmentation as a text generation problem, eliminating the need for additional decoders and significantly simplifying the segmentation process. Our key innovation is semantic descriptors, a new textual representation of segmentation masks where each image patch is mapped to its corresponding text label. This unified representation allows seamless integration into the auto-regressive training pipeline of MLLMs for easier optimization. We demonstrate that representing an image with 16×16 semantic descriptors yields competitive segmentation performance. To enhance efficiency, we introduce the Row-wise Run-Length Encoding (R-RLE), which compresses redundant text sequences, reducing the length of semantic descriptors by 74% and accelerating inference by 3×, without compromising performance. Extensive experiments across various vision tasks, such as referring expression segmentation and comprehension, show that Text4Seg achieves state-of-the-art performance on multiple datasets by fine-tuning different MLLM backbones. Our approach provides an efficient, scalable solution for vision-centric tasks within the MLLM framework.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 62 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

56 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference17 Jul
  2. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb

Other references · 54

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  2. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023A Survey on Multimodal Large Language Modelscited by 898
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  8. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  13. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  14. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  15. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  17. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  18. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  20. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  21. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  22. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  23. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  24. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  25. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  26. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  27. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  28. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  29. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  30. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  31. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 132
  32. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  33. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  34. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  35. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  36. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  37. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  38. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 27; 2026: 30.

59 papers.

Sort:

By area:

  1. 2024GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Groundingcited by 46
  2. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 40
  3. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  4. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  5. 2025Multimodal Referring Segmentation: A Surveycited by 22
  6. 2025Affordance-R1: Reinforcement Learning for Generalizable Affordance Reasoning in Multimodal Large Language Modelcited by 21
  7. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  8. 2025OmniAD: Detect and Understand Industrial Anomaly via Multimodal Reasoningcited by 18
  9. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  10. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  11. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  12. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
38 more
  1. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  2. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  3. 2025Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decodercited by 7
  4. 2025Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Predictioncited by 7
  5. 2025ShelfOcc: Native 3D Supervision beyond LiDAR for Vision-Based Occupancy Estimationcited by 6
  6. 2025VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Predictioncited by 6
  7. 2026TextShield-R1: Reinforced Reasoning for Tampered Text Detectioncited by 6
  8. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  9. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 5
  10. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  11. 2025Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Datasetcited by 5
  12. 2025VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learningcited by 5
  13. 2026Towards Pixel-Level VLM Perception via Simple Points Predictioncited by 5
  14. 2026Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Groundingcited by 5
  15. 2026Vision as Unified Multimodal Generationcited by 5
  16. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  17. 2025LlamaSeg: Image Segmentation via Autoregressive Mask Generationcited by 4
  18. 2026JUDO: A Juxtaposed Domain-Oriented Multimodal Reasoner for Industrial Anomaly QAcited by 4
  19. 2025ViLaCD-R1: A Vision-Language Framework for Semantic Change Detection in Remote Sensingcited by 3
  20. 2025Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Maskscited by 2
  21. 2025UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretationcited by 2
  22. 2026Multimodal Large Language Models for Multi-Subject In-Context Image Generationcited by 2
  23. 2026SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Predictioncited by 2
  24. 2025CoSimGen: Controllable Diffusion Model for Simultaneous Image and Mask Generationcited by 1
  25. 2026BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Groundingcited by 1
  26. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentationcited by 1
  27. 2026IQA-Spider: Unifying Multi-Granularity Image Quality Assessment with Reasoning, Grounding and Referringcited by 1
  28. 2026From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Modelscited by 1
  29. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  30. 2025GLACIA: Instance-Aware Positional Reasoning for Glacial Lake Segmentation via Multimodal Large Language Modelcited by 0
  31. 2025Moment and Highlight Detection via MLLM Frame Segmentationcited by 0
  32. 2026VIPA: Visual Informative Part Attention for Referring Image Segmentationcited by 0
  33. 2026Moondream Segmentation: From Words to Maskscited by 0
  34. 2026MAgSeg: Segmentation of Agricultural Landscapes in High-Resolution Satellite Imagery using Multimodal Large Language Modelscited by 0
  35. 2026Self-Improving Small Object Grounding in LVLMscited by 0
  36. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  37. 2026PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Drivingcited by 0
  38. 2026OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observationcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
  2. 2025Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation
  3. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
  4. 2023IFSeg: Image-free Semantic Segmentation via Vision-Language Model
  5. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
  6. 2024MTA-CLIP: Language-Guided Semantic Segmentation with Mask-Text Alignment
  7. 2025GS: Generative Segmentation via Label Diffusion
  8. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models