mlsanityData to 14 Oct 2026

A spatiotemporal style transfer algorithm for dynamic visual stimulus generation

arXiv 2403.04940 ↗First posted 7 Mar 2024cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Understanding how visual information is encoded in biological and artificial systems often requires vision scientists to generate appropriate stimuli to test specific hypotheses. Although deep neural network models have revolutionized the field of image generation with methods such as image style transfer, available methods for video generation are scarce. Here, we introduce the Spatiotemporal Style Transfer (STST) algorithm, a dynamic visual stimulus generation framework that allows powerful manipulation and synthesis of video stimuli for vision research. It is based on a two-stream deep neural network model that factorizes spatial and temporal features to generate dynamic visual stimuli whose model layer activations are matched to those of input videos. As an example, we show that our algorithm enables the generation of model metamers, dynamic stimuli whose layer activations within our two-stream model are matched to those of natural videos. We show that these generated stimuli match the low-level spatiotemporal features of their natural counterparts but lack their high-level semantic features, making it a powerful paradigm to study object recognition. Late layer activations in deep vision models exhibited a lower similarity between natural and metameric stimuli compared to early layers, confirming the lack of high-level information in the generated stimuli. Finally, we use our generated stimuli to probe the representational capabilities of predictive coding deep networks. These results showcase potential applications of our algorithm as a versatile tool for dynamic stimulus generation in vision science.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 106 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

37 of 106 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  12. 2017The Kinetics Human Action Video Datasetcited by 1,909
25 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  3. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  4. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  6. 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
  7. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  8. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  9. 2022Kubric: A scalable dataset generatorcited by 381
  10. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  11. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  12. 2017Neural Style Transfer: A Reviewcited by 199
  13. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  14. 2016Texture Networks: Feed-forward Synthesis of Textures and Stylized Imagescited by 174
  15. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  16. 2016Controlling Perceptual Factors in Neural Style Transfercited by 100
  17. 2016Spatiotemporal Residual Networks for Video Action Recognitioncited by 91
  18. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 77
  19. 2014Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognitioncited by 50
  20. 2017Artistic style transfer for videos and spherical imagescited by 45
  21. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  22. 2017Two-Stream Convolutional Networks for Dynamic Texture Synthesiscited by 15
  23. 2020Crowding Reveals Fundamental Differences in Local vs. Global Processing in Humans and Machinescited by 9
  24. 2017Synthesising Dynamic Textures using Convolutional Neural Networkscited by 6
  25. 2023Divergences in Color Perception between Deep Neural Networks and Humanscited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Geometric Origins of Bias in Deep Neural Networks: A Human Visual System Perspectivecited by 0
  2. 2025Sensory robustness through top-down feedback and neural stochasticity in recurrent vision modelscited by 0