# Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs arXiv: https://arxiv.org/abs/2606.01215 ยท HTML: https://mlsanity.com/paper/2606.01215/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. - Builds on 44 of its 58 references that are in the map. - First posted on arXiv 31 May 2026. ## Placed in - LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/) - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 30 of 44) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023) - [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025) - [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019) - [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023) - [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022) - [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018) - [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022) - [R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization](https://mlsanity.com/paper/2503.12937/) (2025) - [LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL](https://mlsanity.com/paper/2503.07536/) (2025) - [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023) - [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020) - [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023) - [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023) - [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024) - [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023) - [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023) - [Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding](https://mlsanity.com/paper/2412.00493/) (2024) - [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024) - [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017) - [Grounded 3D-LLM with Referent Tokens](https://mlsanity.com/paper/2405.10370/) (2024) - [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023) - [Unifying 3D Vision-Language Understanding via Promptable Queries](https://mlsanity.com/paper/2405.11442/) (2024) - [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025) - [NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations](https://mlsanity.com/paper/2303.13483/) (2023) - [Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning](https://mlsanity.com/paper/2503.00513/) (2025) - [3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding](https://mlsanity.com/paper/2507.23478/) (2025) ## Built on by (most cited first, 2 of 2) - [Explainable Deepfake Detection with Feature-robust Augmentation and Evidence-grounded Explanation Optimization](https://mlsanity.com/paper/2608.20913/) (2026) - [NeuSOGA3D: A Neuro-Symbolic Framework for Explainable 3D Geometric Reconstruction](https://mlsanity.com/paper/2609.20323/) (2026) ## Abstract (arXiv) Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification. We introduce APEIRIA, a neuro-symbolic 3D MLLM to bridge two paradigms by distilling symbolic reasoning patterns into MLLMs with natural language chain-of-thought. Our three-stage curriculum progressively builds reasoning capabilities: a) 3D perception alignment grounds object visual-geometric features to the LLM, b) CoT-SFT teaches query decomposition and stepwise verification from symbolic program traces, and c) CoT-RL extends reasoning patterns to open-set concepts and deeply nested instructions. By transferring reasoning patterns rather than concept-specific knowledge, APEIRIA preserves key NS3D virtues: transparent reasoning and modular interchangeability of planning and perception components. Evaluations on grounding, question answering, and captioning show that APEIRIA surpasses prior NS3D methods and matches state-of-the-art 3D MLLMs on 3D spatial reasoning datasets, unifying symbolic methods' systematic reasoning with MLLMs' flexibility. Code is available at https://github.com/oceanflowlab/APEIRIA.