mlsanityData to 14 Oct 2026

LLaFS: When Large Language Models Meet Few-Shot Segmentation

arXiv 2311.16926 ↗First posted 28 Nov 2023cs.CVcited by 48

Not on the map. Why?

Abstract (arXiv)

This paper proposes LLaFS, the first attempt to leverage large language models (LLMs) in few-shot segmentation. In contrast to the conventional few-shot segmentation methods that only rely on the limited and biased information from the annotated support images, LLaFS leverages the vast prior knowledge gained by LLM as an effective supplement and directly uses the LLM to segment images in a few-shot manner. To enable the text-based LLM to handle image-related tasks, we carefully design an input instruction that allows the LLM to produce segmentation results represented as polygons, and propose a region-attribute table to simulate the human visual mechanism and provide multi-modal guidance. We also synthesize pseudo samples and use curriculum learning for pretraining to augment data and achieve better optimization. LLaFS achieves state-of-the-art results on multiple datasets, showing the potential of using LLMs for few-shot computer vision tasks.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

61 of 79 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  11. 2023A Survey of Large Language Modelscited by 2,576
  12. 2016Pyramid Scene Parsing Networkcited by 2,479
38 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  5. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2023Instruction Tuning with GPT-4cited by 686
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  9. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  10. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  11. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  12. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  13. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  14. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  15. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  16. 2021Curriculum Learning: A Surveycited by 259
  17. 2019CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learningcited by 217
  18. 2021Adaptive Prototype Learning and Allocation for Few-Shot Segmentationcited by 207
  19. 2019Feature Weighting and Boosting for Few-Shot Segmentationcited by 189
  20. 2019PolarMask: Single Shot Instance Segmentation with Polar Representationcited by 188
  21. 2020Part-aware Prototype Network for Few-shot Semantic Segmentationcited by 162
  22. 2022Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentationcited by 152
  23. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  24. 2020Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?cited by 135
  25. 2022Learning What Not to Segment: A New Perspective on Few-Shot Segmentationcited by 120
  26. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  27. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  28. 2020CRNet: Cross-Reference Networks for Few-Shot Segmentationcited by 101
  29. 2022Self-Support Few-Shot Semantic Segmentationcited by 96
  30. 2023Hierarchical Dense Correlation Distillation for Few-Shot Segmentationcited by 84
  31. 2021Mining Latent Classes for Few-shot Segmentationcited by 69
  32. 2022Learning Non-target Knowledge for Few-shot Semantic Segmentationcited by 57
  33. 2021Learning Statistical Texture for Semantic Segmentationcited by 51
  34. 2023Continual Semantic Segmentation with Automatic Memory Sample Selectioncited by 39
  35. 2023Structural and Statistical Texture Knowledge Distillation for Semantic Segmentationcited by 36
  36. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  37. 2023Self-Calibrated Cross Attention Network for Few-Shot Segmentationcited by 32
  38. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 18; 2026: 8.

48 papers.

Sort:

By area:

  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  2. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  3. 2024SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and Morecited by 60
  4. 2024LLMs are Good Action Recognizerscited by 37
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  6. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  7. 2024LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Contentcited by 33
  8. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  9. 2024Hybrid Mamba for Few-Shot Segmentationcited by 21
  10. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understandingcited by 20
  11. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  12. 2024Eliminating Feature Ambiguity for Few-Shot Segmentationcited by 19
36 more
  1. 2024Discrete Latent Perspective Learning for Segmentation and Detectioncited by 16
  2. 2024RESMatch: Referring Expression Segmentation in a Semi-Supervised Mannercited by 11
  3. 2025Unlocking the Power of SAM 2 for Few-Shot Segmentationcited by 11
  4. 2024View-Centric Multi-Object Tracking with Homographic Matching in Moving UAVcited by 10
  5. 2024PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentationcited by 10
  6. 2025Structural and Statistical Texture Knowledge Distillation and Learning for Segmentationcited by 9
  7. 2023ChangeNet: Multi-Temporal Asymmetric Change Detection Datasetcited by 8
  8. 2024Empowering Segmentation Ability to Multi-modal Large Language Modelscited by 8
  9. 2025SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentationcited by 7
  10. 2025Multi-Perspective Data Augmentation for Few-shot Object Detectioncited by 6
  11. 2025Patient-Level Anatomy Meets Scanning-Level Physics: Personalized Federated Low-Dose CT Denoising Empowered by Large Language Modelcited by 6
  12. 2025DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentationcited by 6
  13. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 6
  14. 2026StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compressioncited by 5
  15. 2024MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentationcited by 4
  16. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMscited by 4
  17. 2025RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoningcited by 4
  18. 2026DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generationcited by 4
  19. 2024Beyond Mask: Rethinking Guidance Types in Few-shot Segmentationcited by 3
  20. 2026HD-VGGT: High-Resolution Visual Geometry Transformercited by 3
  21. 2024PerSense: Training-Free Personalized Instance Segmentation in Dense Imagescited by 2
  22. 2024Cross-Modal Few-Shot Learning: a Generative Transfer Learning Frameworkcited by 2
  23. 2025Towards PerSense++: Advancing Training-Free Personalized Instance Segmentation in Dense Imagescited by 2
  24. 2026V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrievalcited by 2
  25. 2026Robust 4D Visual Geometry Transformer with Uncertainty-Aware Priorscited by 2
  26. 2024AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentationcited by 1
  27. 2025ViRefSAM: Visual Reference-Guided Segment Anything Model for Remote Sensing Segmentationcited by 1
  28. 2025A Comparison and Evaluation of Fine-tuned Convolutional Neural Networks to Large Language Models for Image Classification and Segmentation of Brain Tumors on MRIcited by 1
  29. 2025Unbiased Semantic Decoding with Vision Foundation Models for Few-shot Segmentationcited by 1
  30. 2024Symmetrical Joint Learning Support-query Prototypes for Few-shot Segmentationcited by 0
  31. 2024Scene Co-pilot: Procedural Text to Video Generation with Human in the Loopcited by 0
  32. 2025Breaking the Box: Enhancing Remote Sensing Image Segmentation with Freehand Sketchescited by 0
  33. 2025Efficient Masked Attention Transformer for Few-Shot Classification and Segmentationcited by 0
  34. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  35. 2026GeoFlow: Geo-Aware Modeling of Inter-Area Relationships in Origin-Destination Flow Prediction and Generationcited by 0
  36. 2026Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentationcited by 0