# TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control

arXiv: https://arxiv.org/abs/2603.09332 · HTML: https://mlsanity.com/paper/2603.09332/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model](https://mlsanity.com/paper/2304.13731/) (2023)
- [Audiobox: Unified Audio Generation with Natural Language Prompts](https://mlsanity.com/paper/2312.15821/) (2023)
- [HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2504.12330/) (2025)
- [Automatic multitrack mixing with a differentiable mixing console of neural audio effects](https://mlsanity.com/paper/2010.10291/) (2020)
- [AudioLCM: Text-to-Audio Generation with Latent Consistency Models](https://mlsanity.com/paper/2406.00356/) (2024)
- [M2D-CLAP: Masked Modeling Duo Meets CLAP for Learning General-purpose Audio-Language Representation](https://mlsanity.com/paper/2406.02032/) (2024)
- [ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds](https://mlsanity.com/paper/2409.09213/) (2024)
- [Audio representations for deep learning in sound synthesis: A review](https://mlsanity.com/paper/2201.02490/) (2022)
- [PALM: Few-Shot Prompt Learning for Audio Language Models](https://mlsanity.com/paper/2409.19806/) (2024)
- [ST-ITO: Controlling Audio Effects for Style Transfer with Inference-Time Optimization](https://mlsanity.com/paper/2410.21233/) (2024)
- [DDSP-SFX: Acoustically-guided sound effects generation with differentiable digital signal processing](https://mlsanity.com/paper/2309.08060/) (2023)
- [Blind estimation of audio effects using an auto-encoder approach and differentiable digital signal processing](https://mlsanity.com/paper/2310.11781/) (2023)
- [Audio Texture Synthesis with Scattering Moments](https://mlsanity.com/paper/1311.0407/) (2013)
- [SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration](https://mlsanity.com/paper/2312.04690/) (2023)
- [WikiMuTe: A web-sourced dataset of semantic descriptions for music audio](https://mlsanity.com/paper/2312.09207/) (2023)
- [Can Large Language Models Predict Audio Effects Parameters from Natural Language?](https://mlsanity.com/paper/2505.20770/) (2025)
- [Automatic Music Mixing using a Generative Model of Effect Embeddings](https://mlsanity.com/paper/2511.08040/) (2025)

## Abstract (arXiv)

Digital audio workstations expose rich effect chains, yet a semantic gap remains between perceptual user intent and low-level signal-processing parameters. We study retrieval-grounded audio effect control, where the output is an editable plugin configuration rather than a finalized waveform. Our focus is Texture Resonance Retrieval (TRR), an audio representation built from Gram matrices of projected mid-level Wav2Vec2 activations. This design preserves texture-relevant co-activation structure. We evaluate TRR on a guitar-effects benchmark with 1,063 candidate presets and 204 queries. The evaluation follows Protocol-A, a cross-validation scheme that prevents train-test leakage. We compare TRR against CLAP and internal retrieval baselines (Wav2Vec-RAG, Text-RAG, FeatureNN-RAG), using min-max normalized metrics grounded in physical DSP parameter ranges. Ablation studies validate TRR's core design choices: projection dimensionality, layer selection, and projection type. A near-duplicate sensitivity analysis confirms that results are robust to trivial knowledge-base matches. TRR achieves the lowest normalized parameter error among evaluated methods. A multiple-stimulus listening study with 26 participants provides complementary perceptual evidence. We interpret these results as benchmark evidence that texture-aware retrieval is useful for editable audio effect control, while broader personalization and real-audio robustness claims remain outside the verified evidence presented here.