mlsanityData to 7 Oct 2026

Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation

arXiv 2609.26182 ↗First posted 10 Aug 2026cs.CL cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outputs. Extending such a model to a new modality with existing parameter-efficient methods silently changes those outputs; LoRA-style adaptation rewrites the text path whether or not the weights are merged, invalidating every stored embedding. We propose modality-gated deep adapters: bottleneck adapters attached to every decoder layer of a frozen multimodal embedding LLM, grouped into per-modality packs that execute only while their own modality is being encoded. The result is a modality added with zero change to existing outputs: inputs no pack claims traverse the base model's own computation graph, bit-for-bit unchanged, and co-loaded packs compose with an exact-zero isolation matrix. Both properties are stated as propositions, hold after arbitrary training rather than only at initialization, require no task labels or routing metadata at inference, and are verified by exact-equality tests on the released checkpoints. On one frozen 2B base, the audio pack (injected as connector tokens) improves audio-to-text R@10 by +3.4 to +5.4 points over an identically trained control, positive at every seed and reproduced at eleven times the data; the thermal pack, reusing the base's own frozen vision path, clears its pre-registered acceptance gate roughly sevenfold at every seed and lifts thermal-to-text R@10 from 0.224 to 0.785. An encoder swap locates the missing capacity: an external audio encoder that outranks Whisper-family encoders in CLAP-style comparisons loses by 16 R@10 points inside the frozen LLM, so the capacity belongs in the layers, exactly where the gated adapters place it. We release the audio model, the thermal pack, and the training, evaluation and invariance suites: models at huggingface.co/EximiusLabs, code on GitHub.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

31 of 38 references.

References · 31

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  7. 2025Qwen2.5-Omni Technical Reportcited by 704
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  10. 2019Clotho: An Audio Captioning Datasetcited by 354
  11. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  12. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
19 more
  1. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  2. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  3. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  4. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  5. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  6. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  7. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  8. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  9. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  10. 2024Scaling up masked audio encoder learning for general audio classificationcited by 43
  11. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  12. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  13. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  14. 2025M2D-CLAP: Exploring General-purpose Audio-Language Representations Beyond CLAPcited by 10
  15. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  16. 2026jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towerscited by 4
  17. 2026Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrievalcited by 3
  18. 2026Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audiocited by 3
  19. 2026Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embeddingcited by 1

Built on by

Not cited yet.