# On the Challenges and Opportunities in Generative AI

arXiv: https://arxiv.org/abs/2403.00025 · HTML: https://mlsanity.com/paper/2403.00025/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 295 of its 391 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 30 of 295)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)

## Built on by (most cited first, 30 of 31)

- [Generative to Agentic AI: Survey, Conceptualization, and Challenges](https://mlsanity.com/paper/2504.18875/) (2025)
- [I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token](https://mlsanity.com/paper/2412.06676/) (2024)
- [FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion](https://mlsanity.com/paper/2506.04648/) (2025)
- [Fast Samplers for Inverse Problems in Iterative Refinement Models](https://mlsanity.com/paper/2405.17673/) (2024)
- [Unity by Diversity: Improved Representation Learning in Multimodal VAEs](https://mlsanity.com/paper/2403.05300/) (2024)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspective](https://mlsanity.com/paper/2507.05914/) (2025)
- [Investigating Wit, Creativity, and Detectability of Large Language Models in Domain-Specific Writing Style Adaptation of Reddit's Showerthoughts](https://mlsanity.com/paper/2405.01660/) (2024)
- [Statistical Methods in Generative AI](https://mlsanity.com/paper/2509.07054/) (2025)
- [Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations](https://mlsanity.com/paper/2511.05613/) (2025)
- [Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization](https://mlsanity.com/paper/2507.08342/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [SetPINNs: Set-based Physics-informed Neural Networks](https://mlsanity.com/paper/2409.20206/) (2024)
- [Rendering-Refined Stable Diffusion for Privacy Compliant Synthetic Data](https://mlsanity.com/paper/2412.06248/) (2024)
- [Contextual Memory Intelligence -- A Foundational Paradigm for Human-AI Collaboration and Reflective Generative AI Systems](https://mlsanity.com/paper/2506.05370/) (2025)
- [Tethering Broken Themes: Aligning Neural Topic Models with Labels and Authors](https://mlsanity.com/paper/2410.18140/) (2024)
- [Pretrained LLMs Learn Multiple Types of Uncertainty](https://mlsanity.com/paper/2505.21218/) (2025)
- [Generative AI as a Pillar for Predicting 2D and 3D Wildfire Spread: Beyond Physics-Based Models and Traditional Deep Learning](https://mlsanity.com/paper/2506.02485/) (2025)
- [A Distributed Generative AI Approach for Heterogeneous Multi-Domain Environments under Data Sharing constraints](https://mlsanity.com/paper/2507.12979/) (2025)
- [OTARo: Once Tuning for All Precisions toward Robust On-Device LLMs](https://mlsanity.com/paper/2511.13147/) (2025)
- [From Reflection to Repair: A Scoping Review of Dataset Documentation Tools](https://mlsanity.com/paper/2602.15968/) (2026)
- [Generative Topological Networks](https://mlsanity.com/paper/2406.15152/) (2024)
- [Progressive Monitoring of Generative Model Training Evolution](https://mlsanity.com/paper/2412.12755/) (2024)
- [AI-Generated Content in Cross-Domain Applications: Research Trends, Challenges and Propositions](https://mlsanity.com/paper/2509.11151/) (2025)
- [Match Chat: Real Time Generative AI and Generative Computing for Tennis](https://mlsanity.com/paper/2509.12592/) (2025)
- [seqme: a Python library for evaluating biological sequence design](https://mlsanity.com/paper/2511.04239/) (2025)
- [Low-Bit, High-Fidelity: Optimal Transport Quantization for Flow Matching](https://mlsanity.com/paper/2511.11418/) (2025)
- [The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraints](https://mlsanity.com/paper/2605.19066/) (2026)
- [Physics-informed Conditional Normalizing Flows for Angles-only Cislunar Orbit Determination](https://mlsanity.com/paper/2606.30936/) (2026)

## Abstract (arXiv)

The field of deep generative modeling has grown rapidly in the last few years. With the availability of massive amounts of training data coupled with advances in scalable unsupervised learning paradigms, recent large-scale generative models show tremendous promise in synthesizing high-resolution images and text, as well as structured data such as videos and molecules. However, we argue that current large-scale generative AI models exhibit several fundamental shortcomings that hinder their widespread adoption across domains. In this work, our objective is to identify these issues and highlight key unresolved challenges in modern generative AI paradigms that should be addressed to further enhance their capabilities, versatility, and reliability. By identifying these challenges, we aim to provide researchers with insights for exploring fruitful research directions, thus fostering the development of more robust and accessible generative AI solutions.