# ChartFormer: A Large Vision Language Model for Converting Chart Images into Tactile Accessible SVGs

arXiv: https://arxiv.org/abs/2405.19117 · HTML: https://mlsanity.com/paper/2405.19117/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 7 of 7)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [ChartLlama: A Multimodal LLM for Chart Understanding and Generation](https://mlsanity.com/paper/2311.16483/) (2023)
- [ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning](https://mlsanity.com/paper/2402.12185/) (2024)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning](https://mlsanity.com/paper/2401.02384/) (2024)
- [Line Graphics Digitization: A Step Towards Full Automation](https://mlsanity.com/paper/2307.02065/) (2023)
- [Chart4Blind: An Intelligent Interface for Chart Accessibility Conversion](https://mlsanity.com/paper/2403.06693/) (2024)

## Built on by (most cited first, 3 of 3)

- [C2: Scalable Auto-Feedback for LLM-based Chart Generation](https://mlsanity.com/paper/2410.18652/) (2024)
- [POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering](https://mlsanity.com/paper/2507.11939/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)

## Abstract (arXiv)

Visualizations, such as charts, are crucial for interpreting complex data. However, they are often provided as raster images, which are not compatible with assistive technologies for people with blindness and visual impairments, such as embossed papers or tactile displays. At the same time, creating accessible vector graphics requires a skilled sighted person and is time-intensive. In this work, we leverage advancements in the field of chart analysis to generate tactile charts in an end-to-end manner. Our three key contributions are as follows: (1) introducing the ChartFormer model trained to convert raster chart images into tactile-accessible SVGs, (2) training this model on the Chart2Tactile dataset, a synthetic chart dataset we created following accessibility standards, and (3) evaluating the effectiveness of our SVGs through a pilot user study with an refreshable two-dimensional tactile display. Our work is publicly available at https://github.com/nsothman/ChartFormer .