Running your own AI assistant used to mean renting GPUs and losing a weekend to driver versions. That’s no longer true. A laptop from the last few years and an afternoon is enough for a private assistant that never sends a byte to anyone.

Here’s how the pieces fit together — and, just as importantly, where the honest limits are.

What you need

  • RAM is the real requirement, not the GPU. The model has to fit in memory. 16 GB opens the door; 32 GB makes it comfortable.
  • About 5–20 GB of disk per model, depending on size and quantization.
  • A GPU helps but isn’t required. Apple Silicon does well because CPU and GPU share memory. On a PC, a discrete GPU with enough VRAM is the fastest path; without one it still runs, just slower.

Step 1 — Pick a runtime

You want something that bundles the inference engine and model downloads together. The mainstream options are Ollama (command-line first, trivially scriptable), LM Studio (a GUI, easiest start), and llama.cpp (the engine most of the others are built on, if you want full control).

All of them expose a local HTTP API, which is what makes the rest of this work.

# Ollama: pull a model and talk to it
ollama pull llama3.1:8b
ollama run llama3.1:8b

Step 2 — Choose a model size honestly

This is where most people go wrong. Bigger is not automatically better if it doesn’t fit in RAM — a model that spills to disk doesn’t run slowly, it runs uselessly slowly.

Model sizeNeeds roughlyGood for
3B~4 GBFast autocomplete, classification, simple summaries
7–8B~8 GBThe sweet spot: chat, drafting, most everyday tasks
13–14B~12 GBNoticeably better reasoning, still laptop-viable
30B+24 GB+Genuinely capable, wants a real GPU

Quantization is the lever that makes this work. A 4-bit quantized model (Q4_K_M and friends) is roughly a quarter the size of the 16-bit original with surprisingly little quality loss. Start there; only move up if you can measure a difference that matters to you.

Step 3 — Put a real interface on it

The terminal is fine for testing, tedious for daily use. Open-source front-ends give you a familiar chat UI, conversation history, and model switching. They connect to the local API your runtime already exposes — no account, no internet.

The rule: if a “local” tool asks you to sign in before it will talk to your own model, you’ve picked the wrong tool.

Step 4 — Point it at your own documents

This is where a local assistant stops being a toy. Retrieval — often called RAG — lets the model answer from your notes, papers or code rather than its own compressed memory.

The pipeline is simpler than the acronym suggests:

  1. Your documents are split into chunks.
  2. Each chunk is turned into an embedding (a vector) by a small local model.
  3. Your question is embedded the same way; the closest chunks are found.
  4. Those chunks are pasted into the prompt as context.

The model isn’t “learning” your files — it’s being handed the relevant pages at question time. That’s why it works on a laptop, and why it stays entirely local.

What it genuinely can’t do

Anyone telling you a local 8B model matches a frontier model is selling something.

  • Deep multi-step reasoning is where the gap is widest and most obvious.
  • Obscure factual recall is weak. Small models compress hard, and what gets squeezed out first is the long tail of facts. This is exactly why retrieval matters.
  • Very long context costs memory quadratically in attention. Big context windows on small hardware are a good way to meet your swap file.

So is it worth it?

For summarising your own documents, drafting, brainstorming, rewriting, classifying, and answering everyday questions? Comfortably yes — and it’s genuinely startling the first time you pull the network cable and nothing changes.

What you trade in raw capability you get back in privacy, zero marginal cost, and no dependency on a company’s pricing page. For a large slice of everyday work, that trade is firmly worth making.