mlsanityData to 14 Oct 2026

Cross-Modal Fine-Tuning: Align then Refine

arXiv 2302.05738 ↗First posted 11 Feb 2023cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other modalities due to a lack of relevant pretrained models. In this work, we propose ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. ORCA adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities. Through extensive experiments, we show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforming a wide range of hand-designed, AutoML, general-purpose, and task-specific methods. We highlight the importance of data alignment via a series of ablation studies and demonstrate ORCA's utility in data-limited regimes.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 61 of its 84 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

61 of 84 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2016Densely Connected Convolutional Networkscited by 6,720
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  11. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  5. 2022Visual Prompt Tuningcited by 1,681
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  8. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  9. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,204
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2022A Generalist Agentcited by 808
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  14. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  15. 2018Deep Visual Domain Adaptation: A Surveycited by 635
  16. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  17. 2014OpenML: networked science in machine learningcited by 526
  18. 2018Spherical CNNscited by 456
  19. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 413
  20. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  21. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 344
  22. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  23. 2019ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernelscited by 339
  24. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  25. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 260
  26. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  27. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  28. 2018A System for Massively Parallel Hyperparameter Tuningcited by 202
  29. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  30. 2020Geometric Dataset Distances via Optimal Transportcited by 158
  31. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  32. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  33. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  34. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  35. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  36. 2020Deep Visual Domain Adaptationcited by 91
  37. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  38. 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 10; 2026: 7.

34 papers.

Sort:

By area:

  1. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 108
  2. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 98
  3. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  4. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 39
  5. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  6. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  7. 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30
  8. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  9. 2023Learning Probabilistic Symmetrization for Architecture Agnostic Equivariancecited by 26
  10. 2024LICO: Large Language Models for In-Context Molecular Optimizationcited by 26
  11. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 24
  12. 2025CAT: Content-Adaptive Image Tokenizationcited by 21
22 more
  1. 2026Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perceptioncited by 18
  2. 2026LatentLens: Revealing Highly Interpretable Visual Tokens in LLMscited by 15
  3. 2024Human-aligned Chess with a Bit of Searchcited by 11
  4. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  5. 2024OmniArch: Building Foundation Model For Scientific Computingcited by 9
  6. 2023Unlocking the Transferability of Tokens in Deep Models for Tabular Datacited by 8
  7. 2024Learning Modality Knowledge Alignment for Cross-Modality Transfercited by 8
  8. 2024Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generationcited by 5
  9. 2024Advection Augmented Convolutional Neural Networkscited by 5
  10. 2023Language Semantic Graph Guided Data-Efficient Learningcited by 4
  11. 2025SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Modelcited by 3
  12. 2025Task-Oriented Multimodal Token Transmission in Resource-Constrained Multiuser Networkscited by 3
  13. 2024What explains the success of cross-modal fine-tuning with ORCA?cited by 2
  14. 2025COSMOS: Predictable and Cost-Effective Adaptation of LLMscited by 1
  15. 2025Enhancing LLMs for Time Series Forecasting via Structure-Guided Cross-Modal Alignmentcited by 1
  16. 2026Rethinking Cross-Modal Fine-Tuning: Optimizing the Interaction Between Feature Alignment and Target Fittingcited by 1
  17. 2026SSDA: Bridging Spectral and Structural Gaps via Dual Adaptation for Vision-Based Time Series Forecastingcited by 1
  18. 2024SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modalitycited by 0
  19. 2025Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEscited by 0
  20. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  21. 2026BrokenBind: Universal Modality Exploration beyond Dataset Boundariescited by 0
  22. 2026DiffPDE: Masked Diffusion Language Models as PDE Solvercited by 0