mlsanityData to 7 Oct 2026

SPoRC-VIST: A Benchmark for Evaluating Generative Natural Narrative in Vision-Language Models

arXiv 2601.01062 ↗First posted 3 Jan 2026cs.LG cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-Language Models (VLMs) have achieved remarkable success in descriptive tasks such as image captioning and visual question answering (VQA). However, their ability to generate engaging, long-form narratives -- specifically multi-speaker podcast dialogues -- remains under-explored and difficult to evaluate. Standard metrics like BLEU and ROUGE fail to capture the nuances of conversational naturalness, personality, and narrative flow, often rewarding safe, repetitive outputs over engaging storytelling. In this work, we present a novel pipeline for end-to-end visual podcast generation, and fine-tune a Qwen3-VL-32B model on a curated dataset of 4,000 image-dialogue pairs. Crucially, we use a synthetic-to-real training strategy: we train on high-quality podcast dialogues from the Structured Podcast Research Corpus (SPoRC) paired with synthetically generated imagery, and evaluate on real-world photo sequences from the Visual Storytelling Dataset (VIST). This rigorous setup tests the model's ability to generalize from synthetic training data to real-world visual domains. We propose a comprehensive evaluation framework that moves beyond textual overlap, and use AI-as-a-judge (Gemini 3 Pro, Claude Opus 4.5, GPT 5.2) and novel style metrics (average turn length, speaker switch rate) to assess quality. Our experiments demonstrate that our fine-tuned 32B model significantly outperforms a 235B base model in conversational naturalness (>80% win rate) and narrative depth (+50% turn length), while maintaining identical visual grounding capabilities (CLIPScore: 20.39).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 27 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

17 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
3 more
  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  2. 2016Visual Storytellingcited by 176
  3. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023TouchStone: Evaluating Vision-Language Models by Language Models
  2. 2025VIST-GPT: Ushering in the Era of Visual Storytelling with LLMs?
  3. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Models
  4. 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models
  5. 2024Personalized Large Vision-Language Models
  6. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models
  7. 2025VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models
  8. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions