The last few years of AI were defined by scale: bigger models, bigger data centres, bigger bills. But a quieter revolution has been running in parallel — one measured not in gigawatts but in milliwatts. Increasingly, the AI that touches your life most often isn’t running in a distant data centre. It’s running on the device in your hand.

The three taxes of cloud AI

Cloud inference is powerful, and for the hardest problems it’s still the only game in town. But it carries three costs that never go away, no matter how good the model gets.

Latency. Every request is a round trip. Even a fast one costs a few hundred milliseconds before the model has thought about anything at all. For a chat window, that’s fine. For autocomplete, live captions, or anything that reacts as you type, it’s fatal — the interaction has to feel instant or it feels broken.

Cost. Every call is billed. That’s survivable for a premium feature used occasionally; it’s ruinous for a feature used constantly by everyone. The economics quietly decide which ideas ship and which die in a planning doc.

Privacy. Your data leaves your device. For a search query, most people shrug. For your photos, your messages, your medical notes or your company’s source code, “we send it to a third party and they promise to behave” is a policy, not a guarantee.

Those three taxes explain why a whole category of obvious features never shipped. Nobody wants a keyboard that phones home for every word.

What actually changed

Three things converged, and none alone would have been enough.

Silicon got specialised. Phones and laptops now ship dedicated neural accelerators — NPUs — built to run matrix maths at a fraction of the power a CPU or GPU would burn. The number that matters stopped being raw speed and became performance per watt, because on a phone the real constraint is heat and battery, not peak FLOPS.

Models got dramatically smaller without collapsing. This is the genuinely surprising part. Through quantization (storing weights at 4 or 8 bits instead of 16), distillation (training a small model to imitate a large one), and better training recipes, a compact model now does what needed something many times larger a few years ago. A 3–8B parameter model that once demanded a server runs comfortably on a laptop — and increasingly on a phone.

The software stack matured. Running a model locally stopped being a research project with a CUDA-shaped learning curve. Runtimes now handle quantized formats, memory mapping and hardware acceleration without you thinking about any of it.

Why this changes the shape of products

Privacy becomes structural, not promised. This is the deepest change. When inference happens locally there is no data to leak, subpoena or sell. You don’t have to trust a privacy policy, because there’s no transfer for the policy to be about. That’s a categorically different kind of guarantee.

It works where the network doesn’t. On a plane, in a lift, in a tunnel, in a country with a hostile or expensive network. Features that degrade to nothing without signal are features people quietly learn not to rely on.

It’s free at the margin. No per-request cost changes what’s worth building. Things that were too expensive to give every user on every keystroke suddenly aren’t. This is the shift that unlocks the most product ideas.

It’s instant. No round trip means the model can sit inside the interaction loop rather than beside it.

What local models still can’t do

The ceiling is real, and worth being straight about:

  • Deep reasoning over long context still favours the big models. A small model is excellent at bounded tasks and gets visibly shakier the longer the chain of thought runs.
  • Broad world knowledge is the first thing compressed away when you shrink a model. Small models invent specifics more readily — which is exactly why serious local setups lean on retrieval, feeding the model your actual documents instead of trusting its memory.
  • Memory is the hard wall. Parameters must fit in RAM. That single constraint, far more than compute, decides what can run on your device.

The hybrid future is the actual future

This isn’t the end of cloud AI, and anyone selling it that way is overreaching. The largest models will live in data centres for years, because the physics of memory and power aren’t negotiable.

What’s emerging instead is a handoff. Your device handles the instant, private, common cases — summarise this, transcribe that, clean up this photo, finish this line — and escalates to the cloud only for the genuinely hard ones. The interesting engineering problem of the next few years isn’t building a bigger model. It’s routing: deciding, in milliseconds, which requests deserve the expensive path.

Done well, you’ll never see the seam. That invisibility is the point — and it’s exactly why on-device AI, rather than the next trillion-parameter headline, may turn out to be the most consequential AI story of the decade.