mlsanityData to 7 Oct 2026

Foundation Models Meet Visualizations: Challenges and Opportunities

arXiv 2310.05771 ↗First posted 9 Oct 2023cs.LG cs.HCcited by 31

Not on the map. Why?

Abstract (arXiv)

Recent studies have indicated that foundation models, such as BERT and GPT, excel in adapting to a variety of downstream tasks. This adaptability has established them as the dominant force in building artificial intelligence (AI) systems. As visualization techniques intersect with these models, a new research paradigm emerges. This paper divides these intersections into two main areas: visualizations for foundation models (VIS4FM) and foundation models for visualizations (FM4VIS). In VIS4FM, we explore the primary role of visualizations in understanding, refining, and evaluating these intricate models. This addresses the pressing need for transparency, explainability, fairness, and robustness. Conversely, within FM4VIS, we highlight how foundation models can be utilized to advance the visualization field itself. The confluence of foundation models and visualizations holds great promise, but it also comes with its own set of challenges. By highlighting these challenges and the growing opportunities, this paper seeks to provide a starting point for continued exploration in this promising avenue.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 88 of its 139 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

88 of 139 references.

References · 88

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
38 more
  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023LIMA: Less Is More for Alignmentcited by 1,080
  7. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  8. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  9. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  10. 2019Green AIcited by 620
  11. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  12. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  13. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  14. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
  15. 2019The What-If Tool: Interactive Probing of Machine Learning Modelscited by 207
  16. 2017Neural Style Transfer: A Reviewcited by 199
  17. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  18. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  19. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  20. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  21. 2021Towards Natural Language Interfaces for Data Visualization: A Surveycited by 58
  22. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  23. 2020A Survey of Visual Analytics Techniques for Machine Learningcited by 56
  24. 2016Towards Better Analysis of Deep Convolutional Neural Networkscited by 51
  25. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  26. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  27. 2018Visual Analytics for Explainable Deep Learningcited by 39
  28. 2017Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspectivecited by 36
  29. 2020OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samplescited by 33
  30. 2021AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualizationcited by 32
  31. 2020Analyzing the Noise Robustness of Deep Neural Networkscited by 31
  32. 2020Calliope: Automatic Visual Data Story Generation from a Spreadsheetcited by 30
  33. 2023How Does Attention Work in Vision Transformers? A Visual Analytics Attemptcited by 29
  34. 2020A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualizationcited by 28
  35. 2023Visual Analytics For Machine Learning: A Data Perspective Surveycited by 25
  36. 2023DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articlescited by 24
  37. 2022Towards Natural Language-Based Visualization Authoringcited by 22
  38. 2023Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Modelcited by 21

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 15; 2026: 5.

31 papers.

Sort:

By area:

  1. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  2. 2024Generative AI for Visualization: State of the Art and Future Directionscited by 23
  3. 2025InfoChartQA: A Benchmark for Multimodal Question Answering on Infographic Chartscited by 16
  4. 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactionscited by 13
  5. 2024Human-Guided Image Generation for Expanding Small-Scale Training Image Datasetscited by 10
  6. 2024Visualization Literacy of Multimodal Large Language Models: A Comparative Studycited by 9
  7. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  8. 2024FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injectioncited by 8
  9. 2025DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoningcited by 8
  10. 2024ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Mapcited by 7
  11. 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMscited by 7
  12. 2024AttributionScanner: A Visual Analytics System for Model Validation with Metadata-Free Slice Findingcited by 6
19 more
  1. 2025Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extractioncited by 6
  2. 2024Structural-Entropy-Based Sample Selection for Efficient and Effective Learningcited by 4
  3. 2025InfoDet: A Dataset for Infographic Element Detectioncited by 4
  4. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  5. 2025VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labelscited by 3
  6. 2024LLM-Assisted Visual Analytics: Opportunities and Challengescited by 2
  7. 2025Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Modelscited by 2
  8. 2026RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systemscited by 2
  9. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Modelscited by 1
  10. 2024RuleExplorer: A Scalable Matrix Visualization for Understanding Tree Ensemble Classifierscited by 1
  11. 2025Interactive Hybrid Rice Breeding with Parametric Dual Projectioncited by 1
  12. 2025Story Ribbons: Reimagining Storyline Visualizations with Large Language Modelscited by 1
  13. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1
  14. 2025VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Modelscited by 1
  15. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  16. 2026Human-Data Interaction, Exploration, and Visualization in the AI Era: Challenges and Opportunitiescited by 0
  17. 2026Show Me the Infographic I Imagine: Intent-Aware Infographic Retrieval for Authoring Supportcited by 0
  18. 2026WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analyticscited by 0
  19. 2026LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventionscited by 0