# Double Mixture: Towards Continual Event Detection from Speech

arXiv: https://arxiv.org/abs/2404.13289 · HTML: https://mlsanity.com/paper/2404.13289/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 20 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Joint Extraction of Events and Entities within a Document Context](https://mlsanity.com/paper/1609.03632/) (2016)
- [Incremental Prompting: Episodic Memory Prompt for Lifelong Event Detection](https://mlsanity.com/paper/2204.07275/) (2022)
- [Towards Lifelong Learning of End-to-end ASR](https://mlsanity.com/paper/2104.01616/) (2021)
- [AISHELL-NER: Named Entity Recognition from Chinese Speech](https://mlsanity.com/paper/2202.08533/) (2022)
- [CL-MASR: A Continual Learning Benchmark for Multilingual ASR](https://mlsanity.com/paper/2310.16931/) (2023)
- [Investigating Pre-trained Audio Encoders in the Low-Resource Condition](https://mlsanity.com/paper/2305.17733/) (2023)
- [Towards Relation Extraction From Speech](https://mlsanity.com/paper/2210.08759/) (2022)
- [Training sound event detection with soft labels from crowdsourced annotations](https://mlsanity.com/paper/2302.14572/) (2023)
- [Towards Event Extraction from Speech with Contextual Clues](https://mlsanity.com/paper/2401.15385/) (2024)

## Built on by (most cited first, 1 of 1)

- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)

## Abstract (arXiv)

Speech event detection is crucial for multimedia retrieval, involving the tagging of both semantic and acoustic events. Traditional ASR systems often overlook the interplay between these events, focusing solely on content, even though the interpretation of dialogue can vary with environmental context. This paper tackles two primary challenges in speech event detection: the continual integration of new events without forgetting previous ones, and the disentanglement of semantic from acoustic events. We introduce a new task, continual event detection from speech, for which we also provide two benchmark datasets. To address the challenges of catastrophic forgetting and effective disentanglement, we propose a novel method, 'Double Mixture.' This method merges speech expertise with robust memory mechanisms to enhance adaptability and prevent forgetting. Our comprehensive experiments show that this task presents significant challenges that are not effectively addressed by current state-of-the-art methods in either computer vision or natural language processing. Our approach achieves the lowest rates of forgetting and the highest levels of generalization, proving robust across various continual learning sequences. Our code and data are available at https://anonymous.4open.science/status/Continual-SpeechED-6461.