This study evaluates and improves the calibration of vision-language models like CLIP for zero-shot inference, proposing a modified temperature scaling method for better generalization.
Abstract: Calibration of deep learning models is crucial to their trustworthiness and safe usage, and as such, has been extensively studied in supervised classification models, with methods rafted to decrease miscalibration. However, there has yet to be a comprehensive study of the calibration of vision-language models that are used for zero-shot inference, like CLIP. We measure calibration across relevant variables like prompt, dataset, and architecture, and find that zero-shot inference with CLIP is miscalibrated. Furthermore, we propose a modified version of temperature scaling that is ligned with the common use cases of CLIP as a zero-shot inference model, and show that a single learned temperature generalizes for each specific CLIP model (defined by a chosen pre-training dataset and architecture) across inference dataset and prompt choice.
Accepted to workshop on the pitfalls of limited data and computation for Trustworthy ML, ICLR 2023.
Bibtex Citation
1@misc{levine2023enablingcalibrationzeroshotinference,2 title={Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models}, 3 author={Will LeVine and Benjamin Pikus and Pranav Raja and Fernando Amat Gil},4 year={2023},5 eprint={2303.12748},6 archivePrefix={arXiv},7 primaryClass={cs.CV},8 url={https://arxiv.org/abs/2303.12748}, 9}