mlsanityData to 14 Oct 2026

A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Models

arXiv 2504.02793 ↗First posted 3 Apr 2025cs.AI cs.CL cs.CYcited by 2

Not on the map. Why?

Abstract (arXiv)

Large artificial intelligence (AI) models have garnered significant attention for their remarkable, often "superhuman", performance on standardized benchmarks. However, when these models are deployed in high-stakes verticals such as healthcare, education, and law, they often reveal notable limitations. For instance, they exhibit brittleness to minor variations in input data, present contextually uninformed decisions in critical settings, and undermine user trust by confidently producing or reproducing inaccuracies. These challenges in applying large models necessitate cross-disciplinary innovations to align the models' capabilities with the needs of real-world applications. We introduce a framework that addresses this gap through a layer-wise abstraction of innovations aimed at meeting users' requirements with large models. Through multiple case studies, we illustrate how researchers and practitioners across various fields can operationalize this framework. Beyond modularizing the pipeline of transforming large models into useful "vertical systems", we also highlight the dynamism that exists within different layers of the framework. Finally, we discuss how our framework can guide researchers and practitioners to (i) optimally situate their innovations (e.g., when vertical-specific insights can empower broadly impactful vertical-agnostic innovations), (ii) uncover overlooked opportunities (e.g., spotting recurring problems across verticals to develop practically useful foundation models instead of chasing benchmarks), and (iii) facilitate cross-disciplinary communication of critical challenges (e.g., enabling a shared vocabulary for AI developers, domain experts, and human-computer interaction scholars). Project webpage: https://gaurav22verma.github.io/vertical-systems-with-large-ai-models/

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 60 of its 104 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

60 of 104 references.

References · 60

  1. 2023Segment Anythingcited by 8,234
  2. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
38 more
  1. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  3. 2025Humanity's Last Examcited by 491
  4. 2021Explaining Neural Scaling Lawscited by 363
  5. 2022Matryoshka Representation Learningcited by 282
  6. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  7. 2021Trustworthy AI: From Principles to Practicescited by 207
  8. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  9. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  10. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  11. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  12. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  13. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  14. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  15. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  16. 2024Towards Conversational Diagnostic AIcited by 79
  17. 2020Interpretability of machine learning based prediction models in healthcarecited by 75
  18. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  19. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 66
  20. 2024Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approachcited by 54
  21. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  22. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
  23. 2023Benefits and Harms of Large Language Models in Digital Mental Healthcited by 40
  24. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  25. 2023Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAIcited by 33
  26. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  27. 2024Evaluating Human-AI Collaboration: A Review and Methodological Frameworkcited by 27
  28. 2024LearnLM: Improving Gemini for Learningcited by 25
  29. 2024Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Spacecited by 22
  30. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
  31. 2024To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Modelscited by 18
  32. 2024Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Usecited by 17
  33. 2024Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaborationcited by 13
  34. 2024BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebookscited by 12
  35. 2024Challenges in Human-Agent Communicationcited by 12
  36. 2023Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sourcescited by 10
  37. 2022Overcoming Language Disparity in Online Content Classification with Multimodal Learningcited by 8
  38. 2023Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understandingcited by 8

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  2. 2025X-Intelligence 3.0: Training and Evaluating Reasoning LLM for Semiconductor Displaycited by 0