# SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

arXiv: https://arxiv.org/abs/2406.09833 · HTML: https://mlsanity.com/paper/2406.09833/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 59 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Hyperbolic Image Segmentation](https://mlsanity.com/paper/2203.05898/) (2022)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering](https://mlsanity.com/paper/2204.09634/) (2022)
- [Pan-Mamba: Effective pan-sharpening with State Space Model](https://mlsanity.com/paper/2402.12192/) (2024)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems](https://mlsanity.com/paper/1809.01703/) (2018)
- [HoroPCA: Hyperbolic Dimensionality Reduction via Horospherical Projections](https://mlsanity.com/paper/2106.03306/) (2021)
- [Temporal Reasoning via Audio Question Answering](https://mlsanity.com/paper/1911.09655/) (2019)
- [Progressive Spatio-temporal Perception for Audio-Visual Question Answering](https://mlsanity.com/paper/2308.05421/) (2023)
- [CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Data](https://mlsanity.com/paper/2111.15037/) (2021)
- [Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline](https://mlsanity.com/paper/2209.12028/) (2022)
- [SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding](https://mlsanity.com/paper/2404.01174/) (2024)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)

## Built on by (most cited first, 7 of 7)

- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings](https://mlsanity.com/paper/2505.18973/) (2025)
- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [End-to-End Multi-Modal Diffusion Mamba](https://mlsanity.com/paper/2510.13253/) (2025)
- [AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering](https://mlsanity.com/paper/2510.18346/) (2025)
- [Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization](https://mlsanity.com/paper/2606.07033/) (2026)
- [Hyperbolic Distillation: Geometry-Guided Cross-Modal Transfer for Robust 3D Object Detection](https://mlsanity.com/paper/2605.09899/) (2026)

## Abstract (arXiv)

The Audio-Visual Question Answering (AVQA) task holds significant potential for applications. Compared to traditional unimodal approaches, the multi-modal input of AVQA makes feature extraction and fusion processes more challenging. Euclidean space is difficult to effectively represent multi-dimensional relationships of data. Especially when extracting and processing data with a tree structure or hierarchical structure, Euclidean space is not suitable as an embedding space. Additionally, the self-attention mechanism in Transformers is effective in capturing the dynamic relationships between elements in a sequence. However, the self-attention mechanism's limitations in window modeling and quadratic computational complexity reduce its effectiveness in modeling long sequences. To address these limitations, we propose SHMamba: Structured Hyperbolic State Space Model to integrate the advantages of hyperbolic geometry and state space models. Specifically, SHMamba leverages the intrinsic properties of hyperbolic space to represent hierarchical structures and complex relationships in audio-visual data. Meanwhile, the state space model captures dynamic changes over time by globally modeling the entire sequence. Furthermore, we introduce an adaptive curvature hyperbolic alignment module and a cross fusion block to enhance the understanding of hierarchical structures and the dynamic exchange of cross-modal information, respectively. Extensive experiments demonstrate that SHMamba outperforms previous methods with fewer parameters and computational costs. Our learnable parameters are reduced by 78.12%, while the average performance improves by 2.53%. Experiments show that our method demonstrates superiority among all current major methods and is more suitable for practical application scenarios.