BlogGenerative AI

Small language models: when small, private and on-device wins

A small language model is one compact enough to run on a laptop, a phone or one modest GPU, and current ones handle classification, extraction, routing and short answers well. They win when data must stay on the device, when the product has to work offline, or when volume makes a large model too expensive. They lose on long reasoning and broad knowledge, so the dependable design is a small model first with a large model for the hard cases.

What counts as a small model now?

There is no official line. When we say small, we mean a model of roughly half a billion to ten billion parameters that runs on hardware you can hold or own: a phone, a laptop, an industrial PC, one mid-range GPU. The models behind the big chat products are far larger and live in data centres.

What has changed is how capable the small end has become. These are the open families worth knowing in late 2026:

Family Maker Small sizes Notes
Gemma 4 Google E2B and E4B (about 2 and 4 billion effective parameters) Apache 2.0, reads images and audio, 128K context
Phi-4-mini Microsoft 3.8 billion MIT licence, 128K context
Qwen3.5 Alibaba 0.8, 2, 4 and 9 billion Open weights, text and vision
Llama 3.2 Meta 1 and 3 billion From 2024, under Meta’s own community licence
SmolLM3 Hugging Face 3 billion Apache 2.0

Google announced Gemma 4 in April 2026 and describes the two edge sizes as able to run completely offline on phones, a Raspberry Pi or a Jetson Orin Nano. Microsoft’s model card for Phi-4-mini lists its intended uses as memory-constrained and latency-bound settings.

There is a second category: models you do not ship at all, because the platform provides them. Chrome exposes Gemini Nano to web pages through its Prompt API, and Apple gives apps access to an on-device model through its Foundation Models framework.

Check the licence before you fall for a benchmark. Apache 2.0 and MIT are simple. Some families carry their own terms, and those terms matter if you embed the model in a product you sell.

What do small models do well?

Small models are good at narrow, well-specified jobs with short inputs and short outputs:

  • Classification. Is this message a complaint, an order or spam? Which of 20 queues should it go to?
  • Extraction. Pull the invoice number, date and total out of this text and return JSON.
  • Routing. Decide whether a request can be handled locally or needs a bigger model or a person.
  • Short answers and rewrites. Summarise this note in two lines. Fix the grammar. Translate a sentence.
  • Offline use. Any of the above, where there is no network.

A generic example. A lift maintenance company has 400 engineers who work in basements and plant rooms with no signal. After each job they dictate a note. An on-device model turns the transcript into a structured job record: fault code, parts used, whether a follow-up visit is needed. Nothing leaves the phone until it is back in coverage, and the engineer checks the filled form before leaving the plant room. A large cloud model might do the task slightly better, and it would not be reachable when it is needed.

The common thread is that the task can be described in a short prompt and checked easily. Constrained output helps a great deal. If the runtime can force the model to produce valid JSON that matches a schema, a small model’s main weakness, wandering off format, mostly disappears.

Quantisation and distillation in plain words

Two techniques explain why a useful model fits on a phone.

Quantisation stores each of the model’s numbers with less precision. A model is billions of numbers, normally kept as 16-bit or 32-bit values. Quantisation rounds them to 8 bits or 4 bits. The file shrinks in proportion and inference gets faster, with some loss of quality. The llama.cpp documentation gives a concrete case: an 8-billion-parameter Llama model is about 15 GiB at 16-bit precision and about 4.6 GiB in a common 4-bit format. Four bits is the usual working point. Below that, quality tends to fall away faster, so test before you go lower.

Distillation trains a small model to imitate a large one. You run the large “teacher” model over many examples of your task, collect its outputs, and train the small “student” on them. The student never becomes as broadly capable as the teacher, but on the one task it was taught it can come close. This is the practical route to a small model that is good at your job in particular: use a large model to label a few thousand of your own examples, check a sample by hand, and fine-tune the small model on the result. Our fine-tuning guide covers the mechanics.

The two combine. Distil or fine-tune first, quantise second, and evaluate after each step against the same test set.

Running in the browser or on a phone

You have three routes, and they differ more in operations than in modelling.

Use the model the platform already has. In Chrome, the Prompt API gives web pages access to Gemini Nano, with web support listed from Chrome 148. The catch is the hardware. At the time of writing, Google’s documentation asks for 22 GB of free storage and either a GPU with more than 4 GB of video memory or a CPU with 16 GB of RAM, and says Chrome on Android and iOS is not yet supported. On Apple devices, the Foundation Models framework offers an on-device model with structured output and tool calling, on devices that support Apple Intelligence. In both cases you ship no model and pay no inference cost, and you accept that you do not control which model version the user has.

Ship your own model inside the app. In a mobile app, runtimes such as Google’s LiteRT-LM and llama.cpp run quantised open models on the device’s CPU, GPU or neural processor. In the browser, Transformers.js runs models through ONNX Runtime, on the CPU by default and on the GPU through WebGPU where the browser supports it. You control the model, and you take on a download of one to several gigabytes, memory pressure on older devices, and testing across a long tail of hardware.

Run it near the user instead of on the user’s device. A small server in a clinic, a factory gateway, or a single GPU in your own cloud account. This gives up the offline property but keeps data inside your boundary and makes the hardware predictable. It overlaps with edge AI, where the model sits next to the sensor.

Whichever route you take, build the feature so that it degrades politely. Some users will have a device that cannot run the model at all.

The privacy and cost arguments

The privacy argument is simple and strong. If the text never leaves the device, there is no transfer to a third party to assess, no retention policy to negotiate and no cross-border question. For health notes, legal drafts, children’s data or anything under a strict contract, that can turn a blocked project into an approved one. It does not remove your obligations for what the app stores or syncs afterwards.

The cost argument needs more care. On the user’s device, inference costs you nothing per request. That matters for features used constantly, such as autocomplete or tagging, where even a cheap hosted model adds up. On your own servers, a small model is only cheaper if the hardware is kept busy. A GPU that sits idle most of the day costs more per answer than a hosted model billed by the token. Before buying anything, price the same workload on the smallest hosted model from a major provider, with caching and batching applied. We walk through that comparison in cutting LLM costs.

There is a latency argument as well. A local model starts answering without a network round trip, which users notice in features that respond as they type.

Where do small models fall short?

The model cards say so themselves and are worth reading. Microsoft’s card for Phi-4-mini says the model lacks the capacity to store much factual knowledge and may state things that are wrong, that it performs worse outside English, and that in function-calling tests it sometimes invented function names or URLs.

That matches what we see:

  • Knowledge. Do not ask a small model open questions about the world. Give it the facts in the prompt.
  • Long instructions. A two-page system prompt with 15 rules will be followed in part. Keep prompts short and move rules into code.
  • Multi-step reasoning. Planning across many steps, or holding a long chain of conditions, degrades quickly.
  • Long inputs. The advertised context window may be 128K tokens, but a phone’s memory and the user’s patience give out far sooner.
  • Languages. Quality outside English and a few other major languages varies a lot between families. Test in the languages your users write.

There is also an operational cost that hosted APIs hide from you. You own the evaluation, the model updates and the support tickets from the one phone model on which everything runs slowly.

The pattern that works: small first, large for hard cases

We rarely recommend a small model alone. The dependable design is a cascade.

  1. The small model handles every request first.
  2. It returns its answer together with a signal of how sure it is. For classification this can be the probability of the chosen label. For extraction it can be a validation check: did every required field parse, do the totals add up?
  3. Confident, valid answers are used directly.
  4. Everything else goes to a large hosted model, or to a person.

Apple’s documentation describes the same split: on-device models for summarisation, extraction and similar tasks, and a server model when you need more reasoning and a larger context.

Two numbers tell you whether the cascade is working: the share of requests that escalate, and the error rate among those that do not. If nearly everything escalates, the small model is not earning its keep. If confident answers are often wrong, your confidence signal is not measuring what you think it is. Tune the threshold on a labelled set, not by feel.

When privacy is the reason for going local, the escalation path needs thought, because sending the hard cases to a cloud model may defeat the point. Options include asking the user for consent at that moment, stripping identifying details first, or escalating to a person.

How to start

Pick one narrow task that is high volume or privacy-sensitive, and collect 200 real examples with the correct output. Run them through a large hosted model and through two small candidates in 4-bit form on a laptop. Runtimes such as Ollama or llama.cpp make this an afternoon’s work.

Look at where the small models fail before you look at the average score. If the failures are concentrated in a recognisable kind of input, a cascade will work. If they are scattered at random, try fine-tuning on your examples, and if that does not help, stay with the hosted model for this task.

Only then decide where it runs. Test on the oldest device you intend to support. If you would like help choosing and fitting a model to a device, small and on-device models are part of the generative AI work we do at WeStack AI.

Questions

Asked often.

What is a small language model?

There is no official threshold. In practice people mean models of roughly half a billion to ten billion parameters that can run on a laptop, a phone or one modest GPU, as opposed to the very large models served from data centres. Current examples include Google's Gemma 4 edge models, Microsoft's Phi-4-mini and Alibaba's small Qwen3.5 models.

Can a small language model run on a phone?

Yes. Models of around two to four billion parameters, quantised to four bits, fit in the memory of recent phones and run fully offline. Apple exposes an on-device model to apps through its Foundation Models framework, and Google supplies runtimes such as LiteRT-LM for Android and iOS. Short, focused tasks work best.

Are small language models accurate enough for business use?

For narrow tasks, often yes: classifying messages, pulling fields out of documents, routing requests, short summaries. They are weaker at long chains of reasoning, broad factual recall and long, complicated instructions. The reliable pattern is to give the small model the common cases and pass anything it is unsure about to a larger model.

Is running a small model on-device cheaper than an API?

Per request, yes, because the user's device does the computing and you pay nothing for inference. The costs move elsewhere: engineering time, testing across devices, model downloads of several gigabytes, and updates. On your own servers, a small model is cheaper only if the hardware is kept busy. Compare against the cheapest hosted model before deciding.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.