# Passing: An Endless Journey through Reconstructed Spacetime with AI-Generated Sound

arXiv: https://arxiv.org/abs/2609.27489 · HTML: https://mlsanity.com/paper/2609.27489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners](https://mlsanity.com/paper/2402.17723/) (2024)
- [FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds](https://mlsanity.com/paper/2407.01494/) (2024)
- [Synchformer: Efficient Synchronization from Sparse Cues](https://mlsanity.com/paper/2401.16423/) (2024)
- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching](https://mlsanity.com/paper/2406.00320/) (2024)
- [Temporally Aligned Audio for Video with Autoregression](https://mlsanity.com/paper/2409.13689/) (2024)
- [Read, Watch and Scream! Sound Generation from Text and Video](https://mlsanity.com/paper/2407.05551/) (2024)
- [Tell What You Hear From What You See -- Video to Audio Generation Through Text](https://mlsanity.com/paper/2411.05679/) (2024)
- [SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyond](https://mlsanity.com/paper/2406.17672/) (2024)
- [SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet](https://mlsanity.com/paper/2505.16195/) (2025)
- ['Studies for': A Human-AI Co-Creative Sound Artwork Using a Real-time Multi-channel Sound Generation Model](https://mlsanity.com/paper/2510.25228/) (2025)

## Abstract (arXiv)

This paper introduces Passing, an interactive audiovisual installation that generates an endless journey from a single continuous monorail-window recording by reconstructing it as a spatiotemporal volume. Rather than replaying the footage linearly, the work resamples its spatial and temporal structure along nonlinear trajectories, producing a continuously passing landscape whose depth, speed, and temporal order become unstable. A camera-based viewer-presence detection system estimates whether a viewer is present in the viewing zone and uses this presence state to influence transitions among rendered video sequences. The resulting video stream is fed into SpecMaskFoley, a real-time video-to-audio synthesis model that generates a synchronized soundscape for the reconfigured image. The model is not used to reconstruct an objectively correct soundtrack, but functions as a speculative listener, proposing a possible auditory interpretation of a world whose conventional spatial and temporal premises have been disrupted. Passing distributes creative agency across the artist, who defines the rules of spacetime reconstruction; the AI model, which interprets the emergent visual flow as sound; and the audience, whose embodied presence influences the audiovisual trajectory. Through this structure, the work investigates how authorship and listening may be negotiated among human intention, machine inference, and audience interpretation. Artwork page: https://ryufurusawa.com/passing