# Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insights

arXiv: https://arxiv.org/abs/2505.10043 · HTML: https://mlsanity.com/paper/2505.10043/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions](https://mlsanity.com/paper/2403.19651/) (2024)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval](https://mlsanity.com/paper/2406.04292/) (2024)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)
- [Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval](https://mlsanity.com/paper/2209.00179/) (2022)
- [ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering](https://mlsanity.com/paper/2405.07001/) (2024)
- [Jina CLIP: Your CLIP Model Is Also Your Text Retriever](https://mlsanity.com/paper/2405.20204/) (2024)
- [jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images](https://mlsanity.com/paper/2412.08802/) (2024)
- [MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin](https://mlsanity.com/paper/2310.14037/) (2023)
- [Effortless Data Exploration with zenvisage: An Expressive and Interactive Visual Analytics System](https://mlsanity.com/paper/1604.03583/) (2016)
- [Snowy: Recommending Utterances for Conversational Visual Analysis](https://mlsanity.com/paper/2110.04323/) (2021)
- [Searching the Visual Style and Structure of D3 Visualizations](https://mlsanity.com/paper/1907.11265/) (2019)
- [Learning Style Similarity for Searching Infographics](https://mlsanity.com/paper/1505.01214/) (2015)
- [Olio: A Semantic Search Interface for Data Repositories](https://mlsanity.com/paper/2307.16396/) (2023)
- [SlopeSeeker: A Search Tool for Exploring a Dataset of Quantifiable Trends](https://mlsanity.com/paper/2402.12214/) (2024)

## Built on by (most cited first, 1 of 1)

- [DataVista: Diagnosing Multimodal LLMs on Data Video Understanding](https://mlsanity.com/paper/2610.11993/) (2026)

## Abstract (arXiv)

Text-to-chart retrieval, enabling users to find relevant charts via natural language queries, has gained significant attention. However, evaluating models in real-world business intelligence (BI) scenarios is challenging, as current benchmarks fail to simulate realistic user queries or test for deep semantic understanding with static chart images.To address this gap, we introduce CRBench, the first real-world BI-sourced benchmark comprising 21,862 charts and 326 queries, utilizing a Target-and-Distractor paradigm to evaluate discriminative retrieval among highly similar candidates. Testing on CRBench reveals that existing methods, which rely primarily on visual features, perform poorly and fail to capture the rich analytical semantics of charts. To address this performance bottleneck, we propose a semantic insights synthesis pipeline that automatically generates three hierarchical levels of insights for charts: visual patterns, statistical properties, and practical applications. Using this pipeline, we produced 207,498 semantic insights for 69,166 charts as training data. By leveraging this data to bridge the gap between natural language query intent and latent visual representations via multi-level semantic supervision, we develop ChartFinder, a specialized model capable of deep cross-model reasoning. Experimental results show ChartFinder significantly outperforms state-of-the-art methods on CRBench, achieving up to 66.9% NDCG@10 for precise queries (an 11.58% improvement) and an average increase of 5% across nearly all metrics for fuzzy queries. This work provides the community with a much-needed benchmark for realistic evaluation and demonstrates a powerful data synthesis paradigm for enhancing a model's semantic understanding of charts.