# RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification

arXiv: https://arxiv.org/abs/2511.15923 · HTML: https://mlsanity.com/paper/2511.15923/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding](https://mlsanity.com/paper/2406.14515/) (2024)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://mlsanity.com/paper/2411.00855/) (2024)
- [Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations](https://mlsanity.com/paper/2403.20312/) (2024)
- [Modeling Caption Diversity in Contrastive Vision-Language Pretraining](https://mlsanity.com/paper/2405.00740/) (2024)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding](https://mlsanity.com/paper/2504.07745/) (2025)
- [VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation](https://mlsanity.com/paper/2411.13281/) (2024)
- [VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model](https://mlsanity.com/paper/2407.06491/) (2024)
- [CReFT-CAD: Boosting Orthographic Projection Reasoning for CAD via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.00568/) (2025)
- [VideoPro: A Visual Analytics Approach for Interactive Video Programming](https://mlsanity.com/paper/2308.00401/) (2023)
- [Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation](https://mlsanity.com/paper/2502.15563/) (2025)
- [iSafetyBench: A video-language benchmark for safety in industrial environment](https://mlsanity.com/paper/2508.00399/) (2025)
- [Industrial Language-Image Dataset (ILID): Adapting Vision Foundation Models for Industrial Settings](https://mlsanity.com/paper/2406.09637/) (2024)
- [Benchmarking Vision-Language and Multimodal Large Language Models in Zero-shot and Few-shot Scenarios: A study on Christian Iconography](https://mlsanity.com/paper/2509.18839/) (2025)
- [Utility-Aware Load Shedding for Real-time Video Analytics at the Edge](https://mlsanity.com/paper/2307.02409/) (2023)
- [Better Language Models Exhibit Higher Visual Alignment](https://mlsanity.com/paper/2410.07173/) (2024)

## Built on by (most cited first, 1 of 1)

- [From Imitation to Intuition: Intrinsic Reasoning for Open-Instance Video Classification](https://mlsanity.com/paper/2603.10300/) (2026)

## Abstract (arXiv)

Vision Language Models (VLMs) are becoming increasingly integral to multimedia understanding; however, they often struggle with domain-specific video classification tasks, particularly in cases with limited data. This stems from a critical textitrationale gap, where sparse domain data is insufficient to bridge the semantic distance between complex spatio-temporal content and abstract classification labels. We propose a two-stage self-improvement paradigm to bridge this gap without new annotations. First, we prompt the VLMs to generate detailed textual rationales for each video, compelling them to articulate the domain-specific logic. The VLM is then fine-tuned on these self-generated rationales, utilizing this intermediate supervision to align its representations with the nuances of the target domain. Second, conventional supervised fine-tuning (SFT) is performed on the task labels, achieving markedly higher effectiveness as a result of the model's pre-acquired domain reasoning. Extensive experiments on diverse datasets demonstrate that our method significantly outperforms direct SFT, validating self-generated rationale as an effective, annotation-efficient paradigm for adapting VLMs to domain-specific video analysis.