What is a vision-language model (VLM)?
A vision-language model (VLM) is a large language model (LLM) that takes images as well as text as input and responds in text. Give it a photo of a damaged shipment and ask what is wrong, and it answers in a sentence. Give it a scanned contract and ask for the termination clause, and it quotes it. One model handles both without a separate optical character recognition (OCR) engine or a task-specific detector.
A VLM pairs a vision encoder with a language model. The encoder splits the image into patches and converts them into tokens the language model can attend to alongside the words of the prompt. Training on huge sets of paired images and captions teaches the model to connect what it sees with words, and instruction tuning then teaches it to follow requests. Open-weight VLMs can be downloaded and run on a company’s own graphics processing units (GPUs), which matters when the images are patient records or production drawings.
VLM compared with classic computer vision
The two approaches suit different jobs, and many deployments use both.
| Aspect | Vision-language model | Classic vision model |
|---|---|---|
| Training data needed | None for a new task; a prompt is enough, with optional fine-tuning | Hundreds to thousands of labeled images per task |
| Output | Free text or structured fields | Labels, boxes, masks, or scores |
| Speed and cost per image | Seconds per image on a GPU | Milliseconds on a CPU or a small edge device |
| Best fit | Documents, varied photos, and questions that need context | High-volume, fixed tasks such as line inspection |
VLM in practice
A commercial property insurer receives claim photos from policyholders. A VLM reviews each photo set, describes the visible damage, flags photos that do not match the claim description, and reads the model and serial plate of a damaged rooftop unit so the adjuster can check its age against the policy. A field service company uses the same approach on technician photos to confirm that a repair was completed and that the safety guard was reinstalled. In both cases a person confirms the model’s draft before anything is paid or closed.
What to watch for
- VLMs hallucinate like any LLM; they may read a serial number that is not there, so verify extracted values against a system of record.
- Counting, precise measurement, and small text remain weak spots, and a classic detector or OCR engine often does better on those.
- Cost per image is far higher than for a classic model, so reserve VLMs for varied, low-volume, high-judgment work.
- Check the resolution the model actually processes, because many downscale images and lose fine print.