# TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understanding arXiv: https://arxiv.org/abs/2506.21393 ยท HTML: https://mlsanity.com/paper/2506.21393/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. - Builds on 30 of its 40 references that are in the map. - First posted on arXiv 26 Jun 2025. ## Placed in - LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/) - LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/) ## Builds on (most cited first, 30 of 30) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022) - [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019) - [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019) - [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023) - [Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context](https://mlsanity.com/paper/2005.00589/) (2020) - [When Do Curricula Work?](https://mlsanity.com/paper/2012.03107/) (2020) - [TableBank: A Benchmark Dataset for Table Detection and Recognition](https://mlsanity.com/paper/1903.01949/) (2019) - [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding](https://mlsanity.com/paper/2501.05452/) (2025) - [CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts](https://mlsanity.com/paper/2405.05949/) (2024) - [Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning](https://mlsanity.com/paper/2408.08640/) (2024) - [ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding](https://mlsanity.com/paper/2409.03277/) (2024) - [Multimodal Table Understanding](https://mlsanity.com/paper/2406.08100/) (2024) - [Parsing Table Structures in the Wild](https://mlsanity.com/paper/2109.02199/) (2021) - [Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table](https://mlsanity.com/paper/2305.07288/) (2023) - [FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains](https://mlsanity.com/paper/2311.09797/) (2023) - [Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE](https://mlsanity.com/paper/2409.17508/) (2024) - [Visual Understanding of Complex Table Structures from Document Images](https://mlsanity.com/paper/2111.07129/) (2021) - [A Survey on Curriculum Learning](https://mlsanity.com/paper/2010.13166/) (2020) - [Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning](https://mlsanity.com/paper/2501.13042/) (2025) - [TURL: Table Understanding through Representation Learning](https://mlsanity.com/paper/2006.14806/) (2020) - [Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning](https://mlsanity.com/paper/2503.22456/) (2025) - [TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism](https://mlsanity.com/paper/2501.11800/) (2025) - [Reliable OOD Virtual Screening with Extrapolatory Pseudo-Label Matching](https://mlsanity.com/paper/2406.01825/) (2024) ## Built on by (most cited first, 7 of 7) - [Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2507.11181/) (2025) - [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025) - [Same Content, Different Representations: A Controlled Study for Table QA](https://mlsanity.com/paper/2509.22983/) (2025) - [TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity](https://mlsanity.com/paper/2605.05955/) (2026) - [PARTAB: Partition-Aware Reasoning with Structured Evidence for Scalable Table Understanding](https://mlsanity.com/paper/2608.24082/) (2026) - [InterTab: Interleaved Visual-Structure Alignment for Multi-Modal Table Reasoning](https://mlsanity.com/paper/2609.32660/) (2026) - [MERIT: Multimodal Evidence Reasoning over Image Tables](https://mlsanity.com/paper/2610.17937/) (2026) ## Abstract (arXiv) Multimodal understanding of tables in real-world contexts is challenging due to the complexity of structure, symbolic density, and visual degradation (blur, skew, watermarking, incomplete structures or fonts, multi-span or hierarchically nested layouts). Existing multimodal large language models (MLLMs) struggle with such WildStruct conditions, resulting in limited performance and poor generalization. To address these challenges, we propose TableMoE, a neuro-symbolic Mixture-of-Connector-Experts (MoCE) architecture specifically designed for robust, structured reasoning over multimodal table data. TableMoE features an innovative Neuro-Symbolic Routing mechanism, which predicts latent semantic token roles (e.g., header, data cell, axis, formula) and dynamically routes table elements to specialized experts (Table-to-HTML, Table-to-JSON, Table-to-Code) using a confidence-aware gating strategy informed by symbolic reasoning graphs. To facilitate effective alignment-driven pretraining, we introduce the large-scale TableMoE-Align dataset, consisting of 1.2M table-HTML-JSON-code quadruples across finance, science, biomedicine and industry, utilized exclusively for model pretraining. For evaluation, we curate and release four challenging WildStruct benchmarks: WMMFinQA, WMMTatQA, WMMTabDialog, and WMMFinanceMath, designed specifically to stress-test models under real-world multimodal degradation and structural complexity. Experimental results demonstrate that TableMoE significantly surpasses existing state-of-the-art models. Extensive ablation studies validate each core component, emphasizing the critical role of Neuro-Symbolic Routing and structured expert alignment. Through qualitative analyses, we further showcase TableMoE's interpretability and enhanced robustness, underscoring the effectiveness of integrating neuro-symbolic reasoning for multimodal table understanding.