Vision language models vs classic computer vision: what to use
Use a trained computer vision model when the task is fixed, the volume is high, or you need exact positions, counts or millisecond responses. Use a vision language model when the inputs vary, the judgement needs reading or reasoning, or you have no labelled data and need an answer this month. Many production systems use both: a detector to find things and a language model to judge them.
Two tools for seeing
Classic computer vision models are trained to do one thing. A classifier puts an image into one of a fixed set of categories. A detector draws boxes around objects from a fixed list. A segmentation model outlines them pixel by pixel. An OCR model turns pixels into characters. Each returns structured output with a confidence score, and each has to be shown examples of your problem before it works on your problem.
A vision language model (VLM) is a language model that also accepts images. You describe the task in words, attach the picture and get text back, which can be JSON if you ask for it. Nothing is trained. The hosted models from Anthropic, OpenAI and Google all do this, and so do open-weight models you can run yourself, such as the Qwen3-VL family, whose 8B model is published under the Apache 2.0 licence.
The two are converging at the edges, which we come to below. Their strengths are still different enough that choosing wrongly costs months.
What are classic models good at?
Speed and cost first. Ultralytics publishes figures for its YOLO26 detectors, released in January 2026. On the standard COCO benchmark at 640 pixel input, the smallest model takes about 39 ms per image on a CPU and under 2 ms on a data centre GPU. That is fast enough to check every item on a conveyor, on hardware that sits beside the line.
Precision second. A detector returns pixel coordinates you can measure with. The same image gives the same answer every time. Counting is a loop over the boxes.
The price is data and rigidity. You need labelled examples: a few hundred images is a common starting point for a narrow task, and the number climbs with every variation in lighting, angle and product. Add a new class and you label and retrain, which means owning a small MLOps pipeline. Show the model something outside its training, such as a new packaging design or a camera moved to a different height, and accuracy drops with no warning in the output.
Check licences as well. Ultralytics YOLO is offered under AGPL-3.0 or a paid enterprise licence, which matters if the model ships inside a closed product. Roboflow’s RF-DETR is Apache 2.0 for its Nano to Large detection models and its segmentation models. For text, PaddleOCR is Apache 2.0 and its current OCR models start at 1.5 million parameters, small enough for a phone.
Where do vision language models help, and fail?
A VLM reads almost anything you put in front of it: a photographed receipt, a dashboard screenshot, a handwritten delivery note, a street scene. It follows rules written in plain language. It can explain why it answered as it did. And it starts working in an afternoon, because there is nothing to train.
Its weaknesses are just as real, and the vendors document them. Anthropic’s vision documentation says coordinate and localisation outputs are approximate, and that counts may not be precisely accurate, especially with large numbers of small objects. OpenAI’s images and vision guide says its models struggle with tasks that need precise spatial localisation, may give approximate counts, and can misread rotated or small text.
Three further costs do not appear on a limitations page.
- It is confidently wrong. A detector that is unsure gives a low score. A language model gives a fluent sentence either way. “There are 14 pallets” reads the same whether there are 14 or 17.
- It is not repeatable by default. Ask twice and you may get two answers. For an audit trail that is awkward.
- It is slower and dearer per image. Responses take from a fraction of a second to several seconds. Images are billed as input tokens: OpenAI’s guide works through an example in which one 1024 by 1024 image costs about 1,229 tokens before any text. At a few hundred images a day nobody notices. At ten frames a second from forty cameras it is not an option.
The line is blurring from both sides
Some VLMs now return positions. Google’s image understanding documentation describes Gemini models returning bounding boxes, with coordinates scaled from 0 to 1000, and segmentation masks. Treat these as useful for cropping and rough layout, and verify them before you measure anything with them.
From the other side, classic models have learned to take words. Meta’s SAM 3 segments every instance of a concept named in a short phrase, such as “red crate”, with no training on your data. Meta is frank about the limits. The model does not support longer, complex phrases, and it struggles with fine-grained concepts outside its training domains, such as medical imagery, unless it is fine-tuned. These open-vocabulary models are a good middle path when you need boxes or masks for something no one has labelled yet.
Which should you use for each task?
| Task | Start with | Why | Watch for |
|---|---|---|---|
| Reading documents (invoices, forms, IDs) | A VLM or a document-parsing model, with a schema and validation rules | Layouts vary endlessly and meaning depends on context | Plausible wrong values. Classic OCR still wins for one fixed format at very high volume |
| Inspecting a production line | A trained detector, segmentation or anomaly model, running at the edge | Fixed task, millisecond budget, every item checked | Drift when products or lighting change |
| Counting | A detector or segmentation model, with the count done in code | Exact and repeatable, with a count you can audit | Occlusion and dense piles. A VLM only for small counts where an error is tolerable |
| Describing a scene (alt text, search, incident notes) | A VLM | Open-ended language is what it is for | Invented detail. Keep descriptions short and factual |
| Checking a photo against a written rule | A VLM, often on a crop | The rule is in words and will change | Inconsistent answers at the boundary. Give it a “cannot tell” option |
| Finding a new kind of object with no labels | An open-vocabulary detector or segmenter | Boxes and masks without training | Niche domains need fine-tuning |
Two rows deserve a note. Document reading is covered in depth in our guide to document AI, where the pipeline around the model matters more than the model. And line inspection almost always means running on local hardware, which we describe in computer vision at the edge and build as part of our IoT and edge AI work.
A quick test settles most cases. If you can write the output as numbers and boxes, and the task will be the same next year, train a model. If you would explain the task to a new employee in a paragraph of prose, start with a VLM.
Hybrids: a detector to find, a language model to judge
Most systems we build use both, in one of four patterns.
Detect, crop, judge. A detector finds the region of interest and the VLM sees only the crop. Small things become large, irrelevant background disappears, and the bill falls because the image is smaller. Anthropic’s guidance on coordinates makes the same point: for fine targets, crop the region and send the crop.
Cascade. A cheap model handles the clear cases and passes the doubtful ones upward. If the detector’s confidence is high, accept. If it sits in the grey band, ask the VLM. If the VLM cannot tell, ask a person. The expensive steps see a small share of traffic, the same routing idea we describe in cutting LLM costs.
Language model as labeller. Use a VLM or an open-vocabulary model to pre-label a few thousand images, have people correct them, and train a small detector on the result. You get the VLM’s flexibility during set-up and the detector’s speed in production. This is the usual path from a prototype that works to a system that scales.
Code does the arithmetic. Never ask a language model to add, measure or compare against thresholds. Have models return structured facts and compute the rest.
A worked example: proof-of-delivery photos
Take a parcel carrier that receives 40,000 proof-of-delivery photographs a day. The rule: the parcel must be visible, at a door or in a named safe place, with no person’s face in the frame.
Sending every photo to a VLM would work and would cost more than it needs to. A hybrid does better.
A small detector on the driver’s phone finds parcels, doors and faces. If no parcel is found, the app asks the driver to retake the picture while still on the doorstep, which only works if the answer arrives within a second. If a face is found, it is blurred on the device. Only the photos where a parcel is present but the setting is unclear go to a VLM, with the rule in words and three allowed answers: compliant, not compliant, cannot tell. The last group goes to a person, and their decisions become test data.
Each tool does the part it is good at. The detector is fast and exact about what is in the frame. The language model handles “is that a porch or a pavement”, which no one wants to label ten thousand examples of.
How do you evaluate before choosing?
Evaluation is the same discipline for both tools, and it is the step teams skip because the first demo looked good.
- Collect 200 to 500 of your own images, taken by the real cameras in the real conditions. Include the awkward ones on purpose: night, glare, clutter, damage.
- Label them with the answers you need. Have two people label a sample, to find out whether humans agree with each other. If they do not, the task definition is the problem.
- Score each candidate the right way for the task: precision and recall per class for detection, exact match per field for reading, the spread of count errors for counting, agreement with the human label for rule checks.
- For a VLM, run every image three times and measure how often the answers differ. Inconsistency on the same input is a result in itself.
- Measure latency and cost per image at your expected volume, not on ten test calls.
- Read the failures one by one. Ten minutes with the wrong answers tells you more than the average does.
- Keep the set and run it again whenever the model, prompt or camera changes. Hosted models are updated under you.
Set the pass mark from the cost of each kind of error before you see the scores. A missed defect and a false reject rarely cost the same. The same habits apply to any model-backed system, and our guide to evaluating AI agents goes further into test sets and regression runs.
How to start
In the first week, gather the test set and label it. Then spend one afternoon on the cheapest experiment available: send the images to a hosted VLM with a carefully written instruction and score the answers. That baseline costs almost nothing and tells you how hard the problem is.
The first decision follows from three questions. Does the answer need exact positions or counts? Is the time budget per image under about a second? Will volume pass a few thousand images a day? Three noes: ship the VLM with a review queue and move on. Any yes: plan for a trained model, and use the VLM to label its training data.
Either way, keep the model behind a thin interface of your own, because you will probably swap it. Designing these pipelines, from labelling to deployment, is part of our computer vision and generative AI work.