# VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models

arXiv: https://arxiv.org/abs/2606.27646 · HTML: https://mlsanity.com/paper/2606.27646/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [An Optical Frontend for a Convolutional Neural Network](https://mlsanity.com/paper/1901.03661/) (2018)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)
- [Deep-learning-driven end-to-end metalens imaging](https://mlsanity.com/paper/2312.02669/) (2023)
- [Compressed Meta-Optical Encoder for Image Classification](https://mlsanity.com/paper/2406.06534/) (2024)
- [End-to-end metasurface inverse design for single-shot multi-channel imaging](https://mlsanity.com/paper/2111.01071/) (2021)
- [Neural-Optic Co-Designed Polarization-Multiplexed Metalens for Compact Computational Spectral Imaging](https://mlsanity.com/paper/2311.15164/) (2023)
- [MOCLIP: A Foundation Model for Large-Scale Nanophotonic Inverse Design](https://mlsanity.com/paper/2511.18980/) (2025)

## Abstract (arXiv)

Conventional machine-vision pipelines typically rely on high-quality optics that produce clean, human-interpretable images, and optical design has therefore been driven by image-level criteria such as resolution, aberration correction, and pixel fidelity. However, such optics are often impractical for size-, cost-, or form-factor-constrained applications, where compact meta-optics offer an attractive alternative but operate under strict physical efficiency limits. We propose CODA, a co-design framework that optimizes a continuous-density meta-optic front-end for frozen-model recognition using differentiable image formation and adjoint-gradient updates of Maxwell-based simulations. CODA directly optimizes the cross-entropy loss of a fixed zero-shot CLIP classifier without learned reconstruction, image signal processing, or image-fidelity auxiliary objectives. In a two-dimensional simulated imaging benchmark on ImageNet-100, CODA improves CLIP ViT-L/14 zero-shot accuracy from 53.75 ± 3.57% with a focal-concentration baseline to 65.41 ± 3.99%. The optimized optics further transfer without re-optimization across CLIP, SigLIP, and DINOv2 on ImageNet-100, CIFAR-100, and Food-101. These results demonstrate that, under constrained meta-optic imaging, downstream recognition can be improved by aligning optical design with frozen vision-model objectives rather than conventional image-formation criteria.