# Towards Event Extraction from Speech with Contextual Clues

arXiv: https://arxiv.org/abs/2401.15385 · HTML: https://mlsanity.com/paper/2401.15385/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 27 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://mlsanity.com/paper/2106.09232/) (2021)
- [MAVEN: A Massive General Domain Event Detection Dataset](https://mlsanity.com/paper/2004.13590/) (2020)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://mlsanity.com/paper/2108.12724/) (2021)
- [Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction](https://mlsanity.com/paper/2203.08308/) (2022)
- [Document-level Event Extraction with Efficient End-to-end Learning of Cross-event Dependencies](https://mlsanity.com/paper/2010.12787/) (2020)
- [SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection](https://mlsanity.com/paper/2110.00275/) (2021)
- [M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2207.00952/) (2022)
- [Audio Event Detection using Weakly Labeled Data](https://mlsanity.com/paper/1605.02401/) (2016)
- [Towards Relation Extraction From Speech](https://mlsanity.com/paper/2210.08759/) (2022)
- [A benchmark of state-of-the-art sound event detection systems evaluated on synthetic soundscapes](https://mlsanity.com/paper/2202.01487/) (2022)

## Built on by (most cited first, 3 of 3)

- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Double Mixture: Towards Continual Event Detection from Speech](https://mlsanity.com/paper/2404.13289/) (2024)
- [Continual Speech Learning with Fused Speech Features](https://mlsanity.com/paper/2506.01496/) (2025)

## Abstract (arXiv)

While text-based event extraction has been an active research area and has seen successful application in many domains, extracting semantic events from speech directly is an under-explored problem. In this paper, we introduce the Speech Event Extraction (SpeechEE) task and construct three synthetic training sets and one human-spoken test set. Compared to event extraction from text, SpeechEE poses greater challenges mainly due to complex speech signals that are continuous and have no word boundaries. Additionally, unlike perceptible sound events, semantic events are more subtle and require a deeper understanding. To tackle these challenges, we introduce a sequence-to-structure generation paradigm that can produce events from speech signals in an end-to-end manner, together with a conditioned generation method that utilizes speech recognition transcripts as the contextual clue. We further propose to represent events with a flat format to make outputs more natural language-like. Our experimental results show that our method brings significant improvements on all datasets, achieving a maximum F1 gain of 10.7%. The code and datasets are released on https://github.com/jodie-kang/SpeechEE.