# Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations

arXiv: https://arxiv.org/abs/2212.01033 · HTML: https://mlsanity.com/paper/2212.01033/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 39 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [Multimodal Metric Learning for Tag-based Music Retrieval](https://mlsanity.com/paper/2010.16030/) (2020)
- [Emotion Embedding Spaces for Matching Music to Stories](https://mlsanity.com/paper/2111.13468/) (2021)
- [Generating Music from Literature](https://mlsanity.com/paper/1403.2124/) (2014)

## Built on by (most cited first, 1 of 1)

- [M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models](https://mlsanity.com/paper/2410.09928/) (2024)

## Abstract (arXiv)

Reading, much like music listening, is an immersive experience that transports readers while taking them on an emotional journey. Listening to complementary music has the potential to amplify the reading experience, especially when the music is stylistically cohesive and emotionally relevant. In this paper, we propose the first fully automatic method to build a dense soundtrack for books, which can play high-quality instrumental music for the entirety of the reading duration. Our work employs a unique text processing and music weaving pipeline that determines the context and emotional composition of scenes in a chapter. This allows our method to identify and play relevant excerpts from the soundtrack of the book's movie adaptation. By relying on the movie composer's craftsmanship, our book soundtracks include expert-made motifs and other scene-specific musical characteristics. We validate the design decisions of our approach through a perceptual study. Our readers note that the book soundtrack greatly enhanced their reading experience, due to high immersiveness granted via uninterrupted and style-consistent music, and a heightened emotional state attained via high precision emotion and scene context recognition.