# ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution

arXiv: https://arxiv.org/abs/2502.00989 · HTML: https://mlsanity.com/paper/2502.00989/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution](https://mlsanity.com/paper/2307.16883/) (2023)
- [On Early Detection of Hallucinations in Factual Question Answering](https://mlsanity.com/paper/2312.14183/) (2023)
- [Layer-of-Thoughts Prompting (LoT): Leveraging LLM-Based Retrieval with Constraint Hierarchies](https://mlsanity.com/paper/2410.12153/) (2024)
- [Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models](https://mlsanity.com/paper/2406.18740/) (2024)

## Built on by (most cited first, 1 of 1)

- [TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning](https://mlsanity.com/paper/2607.14658/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.