mlsanityData to 7 Oct 2026

ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding

arXiv 2409.03277 ↗First posted 5 Sep 2024cs.AI cs.CL cs.CVcited by 44

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Automatic chart understanding is crucial for content comprehension and document parsing. Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding through domain-specific alignment and fine-tuning. However, current MLLMs still struggle to provide faithful data and reliable analysis only based on charts. To address it, we propose ChartMoE, which employs the Mixture of Expert (MoE) architecture to replace the traditional linear projector to bridge the modality gap. Specifically, we train several linear connectors through distinct alignment tasks, which are utilized as the foundational initialization parameters for different experts. Additionally, we introduce ChartMoE-Align, a dataset with nearly 1 million chart-table-JSON-code quadruples to conduct three alignment tasks (chart-table/JSON/code). Combined with the vanilla connector, we initialize different experts diversely and adopt high-quality knowledge learning to further refine the MoE connector and LLM parameters. Extensive experiments demonstrate the effectiveness of the MoE connector and our initialization strategy, e.g., ChartMoE improves the accuracy of the previous state-of-the-art from 80.48% to 84.64% on the ChartQA benchmark.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 55 of its 66 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

55 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints9 Dec
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  13. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  14. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  15. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  16. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  17. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  18. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  21. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  22. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  23. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  24. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  25. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149
  26. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  27. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  28. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  29. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  30. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
  31. 2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
  32. 2021SciCap: Generating Captions for Scientific Figurescited by 76
  33. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  34. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  35. 2024OneChart: Purify the Chart Structural Extraction via One Auxiliary Tokencited by 39
  36. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  37. 2024Chart-based Reasoning: Transferring Capabilities from LLMs to VLMscited by 20
  38. 2024ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarizationcited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 21.

44 papers.

Sort:

By area:

  1. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  2. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  3. 2025Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasonercited by 24
  4. 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generationcited by 18
  5. 2026HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Bettercited by 14
  6. 2025PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Imagescited by 11
  7. 2025ChartEditor: A Reinforcement Learning Framework for Robust Chart Editingcited by 9
  8. 2025VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstructioncited by 9
  9. 2025UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesiscited by 8
  10. 2025Shadow-FT: Tuning Instruct Model via Training on Paired Base Modelcited by 7
  11. 2025Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experiencecited by 7
  12. 2026SketchVL: Policy Optimization via Fine-Grained Credit Assignment for Chart Understanding and Morecited by 7
32 more
  1. 2025TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understandingcited by 6
  2. 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
  3. 2025ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understandingcited by 5
  4. 2024Few-Shot Image Quality Assessment via Adaptation of Vision-Language Modelscited by 4
  5. 2025Visual Programmability: A Guide for Code-as-Thought in Chart Understandingcited by 4
  6. 2025START: Spatial and Textual Learning for Chart Understandingcited by 4
  7. 2025ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoningcited by 4
  8. 2026Infinity-Parser2 Technical Reportcited by 4
  9. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentationcited by 3
  10. 2025Chain of Functions: A Programmatic Pipeline for Fine-Grained Chart Reasoning Datacited by 2
  11. 2025In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understandingcited by 2
  12. 2026ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Chartscited by 2
  13. 2026Aligned Multi-View Scripts for Universal Chart-to-Code Generationcited by 2
  14. 2026ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formatscited by 2
  15. 2026ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinementcited by 2
  16. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  17. 2025IE-Critic-R1: Advancing the Explanatory Measurement of Text-Driven Image Editing for Human Perception Alignmentcited by 1
  18. 2026Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answeringcited by 1
  19. 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understandingcited by 1
  20. 2026Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboardscited by 1
  21. 2026LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoningcited by 1
  22. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  23. 2025CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generationcited by 0
  24. 2026Chart Deep Research in LVLMs via Parallel Relative Policy Optimizationcited by 0
  25. 2026CharTool: Tool-Integrated Visual Reasoning for Chart Understandingcited by 0
  26. 2026StructBreak: Structural Cognitive Overload-Induced Safety Failures in MLLMscited by 0
  27. 2026ChartAct: A Benchmark for Dynamic Chart Understandingcited by 0
  28. 2026Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacycited by 0
  29. 2026Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoningcited by 0
  30. 2026What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generationcited by 0
  31. 2026REChart: Reasoning-Efficient Chart Editing with Large Reasoning Modelscited by 0
  32. 2026Back2Struct: Making Structured Images Editable Againcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding
  2. 2025ChartAnchor: Chart Grounding with Structural-Semantic Fidelity
  3. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models
  4. 2024On Pre-training of Multimodal Language Models Customized for Chart Understanding
  5. 2026ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation
  6. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
  7. 2025ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
  8. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs