# Explore In-Context Learning for 3D Point Cloud Understanding

arXiv: https://arxiv.org/abs/2306.08659 · HTML: https://mlsanity.com/paper/2306.08659/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 45 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Masked Autoencoders for Point Cloud Self-supervised Learning](https://mlsanity.com/paper/2203.06604/) (2022)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)

## Built on by (most cited first, 30 of 40)

- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [OMG-Seg: Is One Model Good Enough For All Segmentation?](https://mlsanity.com/paper/2401.10229/) (2024)
- [Point Cloud Mamba: Point Cloud Learning via State Space Model](https://mlsanity.com/paper/2403.00762/) (2024)
- [X-SAM: From Segment Anything to Any Segmentation](https://mlsanity.com/paper/2508.04655/) (2025)
- [Towards Global Optimal Visual In-Context Learning Prompt Selection](https://mlsanity.com/paper/2405.15279/) (2024)
- [DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency](https://mlsanity.com/paper/2504.12080/) (2025)
- [DG-PIC: Domain Generalized Point-In-Context Learning for Point Cloud Understanding](https://mlsanity.com/paper/2407.08801/) (2024)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)
- [VG4D: Vision-Language Model Goes 4D Video Recognition](https://mlsanity.com/paper/2404.11605/) (2024)
- [PCoTTA: Continual Test-Time Adaptation for Multi-Task Point Cloud Understanding](https://mlsanity.com/paper/2411.00632/) (2024)
- [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://mlsanity.com/paper/2403.18660/) (2024)
- [Stable Diffusion Models are Secretly Good at Visual In-Context Learning](https://mlsanity.com/paper/2508.09949/) (2025)
- [OV-VG: A Benchmark for Open-Vocabulary Visual Grounding](https://mlsanity.com/paper/2310.14374/) (2023)
- [Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning](https://mlsanity.com/paper/2504.21263/) (2025)
- [MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing](https://mlsanity.com/paper/2411.16773/) (2024)
- [Point-In-Context: Understanding Point Cloud via In-Context Learning](https://mlsanity.com/paper/2404.12352/) (2024)
- [Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning](https://mlsanity.com/paper/2405.12217/) (2024)
- [How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?](https://mlsanity.com/paper/2511.05449/) (2025)
- [NOLO: Navigate Only Look Once](https://mlsanity.com/paper/2408.01384/) (2024)
- [ConText: Driving In-context Learning for Text Removal and Segmentation](https://mlsanity.com/paper/2506.03799/) (2025)
- [Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning](https://mlsanity.com/paper/2508.10897/) (2025)
- [EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning](https://mlsanity.com/paper/2601.19850/) (2026)
- [PointDGMamba: Domain Generalization of Point Cloud Classification via Generalized State Space Model](https://mlsanity.com/paper/2408.13574/) (2024)
- [Towards Predicting Any Human Trajectory In Context](https://mlsanity.com/paper/2506.00871/) (2025)
- [VIOLA: Towards Video In-Context Learning with Minimal Annotations](https://mlsanity.com/paper/2601.15549/) (2026)
- [PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignment](https://mlsanity.com/paper/2603.18891/) (2026)
- [Physics-Driven Local-Whole Elastic Deformation Modeling for Point Cloud Representation Learning](https://mlsanity.com/paper/2505.13812/) (2025)

## Abstract (arXiv)

With the rise of large-scale models trained on broad data, in-context learning has become a new learning paradigm that has demonstrated significant potential in natural language processing and computer vision tasks. Meanwhile, in-context learning is still largely unexplored in the 3D point cloud domain. Although masked modeling has been successfully applied for in-context learning in 2D vision, directly extending it to 3D point clouds remains a formidable challenge. In the case of point clouds, the tokens themselves are the point cloud positions (coordinates) that are masked during inference. Moreover, position embedding in previous works may inadvertently introduce information leakage. To address these challenges, we introduce a novel framework, named Point-In-Context, designed especially for in-context learning in 3D point clouds, where both inputs and outputs are modeled as coordinates for each task. Additionally, we propose the Joint Sampling module, carefully designed to work in tandem with the general point sampling operator, effectively resolving the aforementioned technical issues. We conduct extensive experiments to validate the versatility and adaptability of our proposed methods in handling a wide range of tasks.