mlsanityData to 14 Oct 2026

Speaking images. A novel framework for the automated self-description of artworks

arXiv 2506.05368 ↗First posted 28 May 2025cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent breakthroughs in generative AI have opened the door to new research perspectives in the domain of art and cultural heritage, where a large number of artifacts have been digitized. There is a need for innovation to ease the access and highlight the content of digital collections. Such innovations develop into creative explorations of the digital image in relation to its malleability and contemporary interpretation, in confrontation to the original historical object. Based on the concept of the autonomous image, we propose a new framework towards the production of self-explaining cultural artifacts using open-source large-language, face detection, text-to-speech and audio-to-animation models. The goal is to start from a digitized artwork and to automatically assemble a short video of the latter where the main character animates to explain its content. The whole process questions cultural biases encapsulated in large-language models, the potential of digital images and deepfakes of artworks for educational purposes, along with concerns of the field of art history regarding such creative diversions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 75 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

34 of 75 references.

References · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  6. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  7. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  8. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  9. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  10. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  11. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  12. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
22 more
  1. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  2. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  3. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  4. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  5. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  6. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  7. 2024Safeguarding Large Language Models: A Surveycited by 78
  8. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 67
  9. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  10. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 36
  11. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 33
  12. 2024GalleryGPT: Analyzing Paintings with Large Multimodal Modelscited by 28
  13. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 27
  14. 2024CulturePark: Boosting Cross-cultural Understanding in Large Language Modelscited by 24
  15. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidancecited by 20
  16. 2024Human Simulacra: Benchmarking the Personification of Large Language Modelscited by 17
  17. 2023What is in a Text-to-Image Prompt: The Potential of Stable Diffusion in Visual Arts Educationcited by 14
  18. 2021Image-to-Video Generation via 3D Facial Dynamicscited by 13
  19. 2023CgT-GAN: CLIP-guided Text GAN for Image Captioningcited by 10
  20. 2024Patterns of Creativity: How User Input Shapes AI-Generated Visual Diversitycited by 4
  21. 2025CognArtive: Large Language Models for Automating Art Analysis and Decoding Aesthetic Elementscited by 3
  22. 2024The Tug-of-War Between Deepfake Generation and Detectioncited by 1

Built on by

Not cited yet.