Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understanding

arXiv 2609.13267 ↗First posted 7 Sep 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more

Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Scientific charts encode quantities in axes, legends, and geometric marks, yet large vision-language models still treat them as natural photographs. Visual in-context examples do not expose the coordinate frame; unconstrained chain-of-thought can name a plausible number that was never read from a bar. We present STEER (Structure-Token Evidence-anchored Reasoning), which freezes a Llama-3.2-Vision encoder and inserts three modules: a chart structure graph encoder (CSGE) that binds ticks, legend items, and marks; evidence-anchored step reasoning (EASR) that forces every arithmetic step to cite a graph node; and weak-parser strong-reasoner alignment (WPSR) that uses a specialized table extractor only as a teacher of node attributes. On ChartQA, STEER reaches 82.70 average relaxed accuracy versus 80.16 for ChartGemma and 76.40 for a LLaVA-CoT backbone trained on the same mix. Gains widen on CharXiv reasoning (33.60 vs. 29.20 InternVL Chat V1.5) and ChartQAPro CoT (40.70 vs. 37.17 Qwen2-VL-7B), where OCR shortcuts disappear. Ablations show that dropping node serialization or numeric candidate constraints undoes most of the reasoning lift.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 51 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

46 of 51 references.

Closest ancestors · 10

Same direction, newest first

Other references · 36

2023GPT-4 Technical Reportcited by 18,396
2024The Llama 3 Herd of Modelscited by 14,715
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2020End-to-End Object Detection with Transformerscited by 7,414
2024GPT-4o System Cardcited by 4,588
2023Improved Baselines with Visual Instruction Tuningcited by 4,418
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079

24 more

2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 234
2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 163
2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
2022Retrieval Augmented Visual Question Answering with Outside Knowledgecited by 94
2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
2024ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wildcited by 79
2025Improving Medical Large Vision-Language Models with Abnormal-Aware Feedbackcited by 42
2024GATEAU: Selecting Influential Samples for Long Context Alignmentcited by 20
2025A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Taskscited by 19
2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memorycited by 14
2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 11
2023Semantic Parsing for Conversational Question Answering over Knowledge Graphscited by 10
2023Multimodal Graph Transformer for Multimodal Question Answeringcited by 10
2025ChartQA-X: Generating Explanations for Visual Chart Reasoningcited by 10
2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way