# BlueGlass: A Framework for Composite AI Safety

arXiv: https://arxiv.org/abs/2507.10106 · HTML: https://mlsanity.com/paper/2507.10106/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 59 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data](https://mlsanity.com/paper/2402.16991/) (2024)
- [A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks](https://mlsanity.com/paper/2306.07303/) (2023)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [Understanding Neural Networks via Feature Visualization: A survey](https://mlsanity.com/paper/1904.08939/) (2019)

## Abstract (arXiv)

As AI systems become increasingly capable and ubiquitous, ensuring the safety of these systems is critical. However, existing safety tools often target different aspects of model safety and cannot provide full assurance in isolation, highlighting a need for integrated and composite methodologies. This paper introduces BlueGlass, a framework designed to facilitate composite AI safety workflows by providing a unified infrastructure enabling the integration and composition of diverse safety tools that operate across model internals and outputs. Furthermore, to demonstrate the utility of this framework, we present three safety-oriented analyses on vision-language models for the task of object detection: (1) distributional evaluation, revealing performance trade-offs and potential failure modes across distributions; (2) probe-based analysis of layer dynamics highlighting shared hierarchical learning via phase transition; and (3) sparse autoencoders identifying interpretable concepts. More broadly, this work contributes foundational infrastructure and findings for building more robust and reliable AI systems.