# Voice2Series: Reprogramming Acoustic Models for Time Series Classification

arXiv: https://arxiv.org/abs/2106.09296 · HTML: https://mlsanity.com/paper/2106.09296/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Transfer learning for time series classification](https://mlsanity.com/paper/1811.01533/) (2018)
- [Sliced Wasserstein Distance for Learning Gaussian Mixture Models](https://mlsanity.com/paper/1711.05376/) (2017)
- [Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition](https://mlsanity.com/paper/2010.13309/) (2020)
- [Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions](https://mlsanity.com/paper/2005.14623/) (2020)
- [A neural attention model for speech command recognition](https://mlsanity.com/paper/1808.08929/) (2018)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals](https://mlsanity.com/paper/1905.11333/) (2019)
- [ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification](https://mlsanity.com/paper/1904.12546/) (2019)
- [Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement](https://mlsanity.com/paper/2003.13917/) (2020)
- [Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentation](https://mlsanity.com/paper/2007.08389/) (2020)
- [Reprogramming Language Models for Molecular Representation Learning](https://mlsanity.com/paper/2012.03460/) (2020)

## Built on by (most cited first, 30 of 91)

- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Foundation Models for Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2403.14735/) (2024)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey](https://mlsanity.com/paper/2302.02515/) (2023)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [The Dawn of Quantum Natural Language Processing](https://mlsanity.com/paper/2110.06510/) (2021)
- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [Evaluation of post-hoc interpretability methods in time-series classification](https://mlsanity.com/paper/2202.05656/) (2022)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Improving Position Encoding of Transformers for Multivariate Time Series Classification](https://mlsanity.com/paper/2305.16642/) (2023)
- [HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models](https://mlsanity.com/paper/2309.15701/) (2023)
- [Data Augmentation techniques in time series domain: A survey and taxonomy](https://mlsanity.com/paper/2206.13508/) (2022)
- [TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation](https://mlsanity.com/paper/2205.12523/) (2022)
- [Visual Prompting for Adversarial Robustness](https://mlsanity.com/paper/2210.06284/) (2022)
- [Towards Neural Scaling Laws for Time Series Foundation Models](https://mlsanity.com/paper/2410.12360/) (2024)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks](https://mlsanity.com/paper/2205.00305/) (2022)
- [Watermarking for Out-of-distribution Detection](https://mlsanity.com/paper/2210.15198/) (2022)
- [Exploring Efficient-tuning Methods in Self-supervised Speech Models](https://mlsanity.com/paper/2210.06175/) (2022)
- [Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting](https://mlsanity.com/paper/2405.14252/) (2024)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [REFINE: Inversion-Free Backdoor Defense via Model Reprogramming](https://mlsanity.com/paper/2502.18508/) (2025)
- [Uncertainty-DTW for Time Series and Sequences](https://mlsanity.com/paper/2211.00005/) (2022)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [Decoding Time Series with LLMs: A Multi-Agent Framework for Cross-Domain Annotation](https://mlsanity.com/paper/2410.17462/) (2024)
- [Reprogramming Pretrained Language Models for Antibody Sequence Infilling](https://mlsanity.com/paper/2210.07144/) (2022)

## Abstract (arXiv)

Learning to classify time series with limited data is a practical yet challenging problem. Current methods are primarily based on hand-designed feature extraction rules or domain-specific data augmentation. Motivated by the advances in deep speech processing models and the fact that voice data are univariate temporal signals, in this paper, we propose Voice2Series (V2S), a novel end-to-end approach that reprograms acoustic models for time series classification, through input transformation learning and output label mapping. Leveraging the representation learning power of a large-scale pre-trained speech processing model, on 30 different time series tasks we show that V2S performs competitive results on 19 time series classification tasks. We further provide a theoretical justification of V2S by proving its population risk is upper bounded by the source risk and a Wasserstein distance accounting for feature alignment via reprogramming. Our results offer new and effective means to time series classification.