mlsanityData to 14 Oct 2026

PerLA: Perceptive 3D Language Assistant

arXiv 2411.19774 ↗First posted 29 Nov 2024cs.CV cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Enabling Large Language Models (LLMs) to understand the 3D physical world is an emerging yet challenging research direction. Current strategies for processing point clouds typically downsample the scene or divide it into smaller parts for separate analysis. However, both approaches risk losing key local details or global contextual information. In this paper, we introduce PerLA, a 3D language assistant designed to be more perceptive to both details and context, making visual representations more informative for the LLM. PerLA captures high-resolution (local) details in parallel from different point cloud areas and integrates them with (global) context obtained from a lower-resolution whole point cloud. We present a novel algorithm that preserves point cloud locality through the Hilbert curve and effectively aggregates local-to-global information via cross-attention and a graph neural network. Lastly, we introduce a novel loss for local representation consensus to promote training stability. PerLA outperforms state-of-the-art 3D language assistants, with gains of up to +1.34 CiDEr on ScanQA for question answering, and +4.22 on ScanRefer and +3.88 on Nr3D for dense captioning. https://gfmei.github.io/PerLA/

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 55 of its 77 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

55 of 77 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  12. 2017Decoupled Weight Decay Regularizationcited by 1,831
38 more
  1. 2021Multiscale Vision Transformerscited by 897
  2. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  3. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  4. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  6. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  8. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  9. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2024What matters when building vision-language models?cited by 326
  12. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  13. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  14. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  15. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  16. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  17. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  18. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  19. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  20. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 167
  21. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 159
  22. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  23. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  24. 2021MPViT: Multi-Path Vision Transformer for Dense Predictioncited by 122
  25. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  26. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 109
  27. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  28. 2021Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusioncited by 72
  29. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  30. 2024When Do We Not Need Larger Vision Models?cited by 67
  31. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  32. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  33. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  34. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  35. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  36. 20213D Question Answeringcited by 41
  37. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  38. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  2. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  3. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  4. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 3
  5. 2026Efficient Encoder-Free Fourier-based 3D Large Multimodal Modelcited by 1
  6. 2026Holo-Captioning: Toward the Text Equivalent of 3D Scenescited by 0
  7. 2026xperception -- Making Robotic Grasping Easiercited by 0
  8. 2026Task-Adaptive Grounded 3D-Programmers Using 2D VLMscited by 0
  9. 2026EgoAlign: Situated 3D Understanding without Situation Descriptionscited by 0