# Judging by Appearances? Auditing and Intervening Vision-Language Models for Bail Prediction

arXiv: https://arxiv.org/abs/2510.00088 · HTML: https://mlsanity.com/paper/2510.00088/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Building and better understanding vision-language models: insights and future directions](https://mlsanity.com/paper/2408.12637/) (2024)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation](https://mlsanity.com/paper/2105.13562/) (2021)
- [Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting](https://mlsanity.com/paper/2407.08223/) (2024)
- [Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration](https://mlsanity.com/paper/2310.09241/) (2023)
- [Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law](https://mlsanity.com/paper/2209.06049/) (2022)
- [Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction](https://mlsanity.com/paper/2307.08321/) (2023)
- [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://mlsanity.com/paper/2407.00600/) (2024)
- [Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning](https://mlsanity.com/paper/2312.05762/) (2023)
- [HLDC: Hindi Legal Documents Corpus](https://mlsanity.com/paper/2204.00806/) (2022)
- [Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models](https://mlsanity.com/paper/2410.10542/) (2024)
- [A Comprehensive Survey on Legal Summarization: Challenges and Future Directions](https://mlsanity.com/paper/2501.17830/) (2025)
- [NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law System](https://mlsanity.com/paper/2508.00709/) (2025)
- [MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages](https://mlsanity.com/paper/2310.09765/) (2023)
- [NyayaAnumana & INLegalLlama: The Largest Indian Legal Judgment Prediction Dataset and Specialized Language Model for Enhanced Decision Analysis](https://mlsanity.com/paper/2412.08385/) (2024)
- [Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice](https://mlsanity.com/paper/2412.15260/) (2024)
- [Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos](https://mlsanity.com/paper/2505.01790/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have been extensively used for legal judgment prediction tasks based on case reports and crime history. However, with a surge in the availability of large vision language models (VLMs), legal judgment prediction systems can now be made to leverage the images of the criminals in addition to the textual case reports/crime history. Applications built in this way could lead to inadvertent consequences and be used with malicious intent. In this work, we run an audit to investigate the efficiency of standalone VLMs in the bail decision prediction task. We observe that the performance is poor across multiple intersectional groups and models textitwrongly deny bail to deserving individuals with very high confidence. We design different intervention algorithms by first including legal precedents through a RAG pipeline and then fine-tuning the VLMs using innovative schemes. We demonstrate that these interventions substantially improve the performance of bail prediction. Our work paves the way for the design of smarter interventions on VLMs in the future, before they can be deployed for real-world legal judgment prediction.