# FLAM: Frame-Wise Language-Audio Modeling

arXiv: https://arxiv.org/abs/2505.05335 · HTML: https://mlsanity.com/paper/2505.05335/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2025.

## Built on by (most cited first, 18 of 18)

- [Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning](https://mlsanity.com/paper/2512.19687/) (2025)
- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding](https://mlsanity.com/paper/2601.06896/) (2026)
- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elements](https://mlsanity.com/paper/2506.09707/) (2025)
- [Event-Grounded Question Answering over Long Audio via Structured Retrieval](https://mlsanity.com/paper/2602.14612/) (2026)
- [Auditory Intelligence: Understanding the World Through Sound](https://mlsanity.com/paper/2508.07829/) (2025)
- [Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?](https://mlsanity.com/paper/2510.14249/) (2025)
- [Steering Where to Listen: Instruction-Based Activation Steering Redirects Temporal Attention in Large Audio-Language Models](https://mlsanity.com/paper/2606.11400/) (2026)
- [ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval](https://mlsanity.com/paper/2512.19703/) (2025)
- [Mix2Morph: Learning Sound Morphing from Noisy Mixes](https://mlsanity.com/paper/2601.20426/) (2026)
- [SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving](https://mlsanity.com/paper/2603.07865/) (2026)
- [Taming Audio VAEs via Target-KL Regularization](https://mlsanity.com/paper/2605.17085/) (2026)
- [CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation](https://mlsanity.com/paper/2605.18916/) (2026)
- [FIGMA: Towards FIne-Grained Music retrievAl](https://mlsanity.com/paper/2606.06615/) (2026)
- [Local Multimodal Music Alignment from Global Supervision](https://mlsanity.com/paper/2607.10023/) (2026)
- [Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA](https://mlsanity.com/paper/2608.11329/) (2026)
- [Does Location Matter? A Black-Box Causal Audit Reveals Model-Dependent Audio Grounding in LALM Hallucination](https://mlsanity.com/paper/2610.13112/) (2026)

## Abstract (arXiv)

Recent multi-modal audio-language models (ALMs) excel at text-audio retrieval but struggle with frame-wise audio understanding. Prior works use temporal-aware labels or unsupervised training to improve frame-wise capabilities, but they still lack fine-grained labeling capability to pinpoint when an event occurs. While traditional sound event detection models can precisely localize events, they are limited to pre-defined categories, making them ineffective for real-world scenarios with out-of-distribution events. In this work, we introduce FLAM, an open-vocabulary contrastive audio-language model capable of localizing specific sound events. FLAM employs a memory-efficient and calibrated frame-wise objective with logit adjustment to address spurious correlations, such as event dependencies and label imbalances during training. To enable frame-wise supervision, we leverage a large-scale dataset with diverse audio events, LLM-generated captions and simulation. Experimental results and case studies demonstrate that FLAM significantly improves the open-vocabulary localization capability while maintaining strong performance in global retrieval and downstream tasks.