mlsanityData to 7 Oct 2026

brat: Aligned Multi-View Embeddings for Brain MRI Analysis

arXiv 2512.18679 ↗First posted 21 Dec 2025cs.CV cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present brat (brain report alignment transformer), a multi-view representation learning framework for brain magnetic resonance imaging (MRI) trained on MRIs paired with clinical reports. Brain MRIs present unique challenges due to the presence of numerous, highly varied, and often subtle abnormalities that are localized to a few slices within a 3D volume. To address these challenges, we introduce a brain MRI dataset 10× larger than existing ones, containing approximately 80,000 3D scans with corresponding radiology reports, and propose a multi-view pre-training approach inspired by advances in document retrieval. We develop an implicit query-feature matching mechanism and adopt concepts from quality-diversity to obtain multi-view embeddings of MRIs that are aligned with the clinical features given by report sentences. We evaluate our approach across multiple vision-language and vision tasks, demonstrating substantial performance improvements. The brat foundation models are publicly released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 21 Dec 2025.

Builds on

43 of 57 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  12. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
31 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2016Colorful Image Colorizationcited by 1,113
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  10. 2021The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classificationcited by 507
  11. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  12. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  13. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  14. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  15. 2019Revealing the Dark Secrets of BERTcited by 300
  16. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 235
  17. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  18. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  19. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  20. 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 124
  21. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  22. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  23. 2022Self-supervised 3D anatomy segmentation using self-distilled masked image transformer (SMIT)cited by 35
  24. 2023Exploring the Boundaries of GPT-4 in Radiologycited by 31
  25. 2020Exploring global diverse attention via pairwise temporal relation for video summarizationcited by 29
  26. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 28
  27. 2023Streaming Active Learning with Deep Neural Networkscited by 20
  28. 2024AMAES: Augmented Masked Autoencoder Pretraining on Public Brain MRI Data for 3D-Native Segmentationcited by 16
  29. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regimecited by 15
  30. 2023Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentationcited by 14
  31. 2024Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentationcited by 8

Built on by

Not cited yet.