Fully Local Voice for Home Assistant, on a CPU

Assumes you run Home Assistant and are comfortable with Docker.

What this does

The house understands us when the internet is down. A 26B model answers from an Unraid box with no GPU, in under three seconds to the first spoken word, on under 1.5 GiB of committed memory. Nothing leaves the building, and it needed no hardware I didn’t already own.

The problem

My voice assistant ran on Claude Haiku: about two seconds to answer, £20–30 a year. The only thing wrong with it was that the house stopped understanding English whenever the broadband went down.

Speech-to-text and text-to-speech were already local. The conversation agent was the last cloud dependency, and the expensive one to replace. Home Assistant hands it a system prompt of roughly 12,700 tokens: tool schemas plus a description of every exposed entity. Before the model can say a word it must read all of it. That read is the prefill, and the state it builds is the KV cache. On a GPU it is fast enough to ignore. On my CPU it takes over two minutes, and most of what follows is about not paying it twice.

The server is an Unraid box: Ryzen 9 7945HX, 64 GB, no usable GPU. The iGPU has two compute units and benchmarks slower than the CPU. Every guide I found assumed otherwise.

The configuration

Gemma 4 26B-A4B, instruction-tuned QAT GGUF, on llama-server. 26B total parameters, ~4B active per token: CPU generation is bandwidth-bound, so an MoE runs at roughly dense-4B speed while answering like something much larger.

Home Assistant reaches it through Local OpenAI LLM by Skye Harris (Apache-2.0), because core HA has no built-in way to point a conversation agent at a local OpenAI-compatible server. Base URL http://<server>:8086/v1, any non-empty string as the API key, the served model name. Then point your pipeline at it and leave “Prefer handling commands locally” enabled.

Five flags do the work:

services:
  llama-server-gemma:
    image: ghcr.io/ggml-org/llama.cpp:full
    network_mode: host
    restart: unless-stopped
    volumes:
      - /mnt/user/appdata/llm-models:/models:ro
      - /mnt/user/appdata/llm-models/slots:/slots
    command:
      - "--server"
      - "-m"
      - "/models/gemma4-26b-a4b-it-qat.gguf"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8086"
      - "-t"
      - "16"              # physical cores, not threads
      - "-c"
      - "16384"
      - "--parallel"
      - "1"
      - "--no-repack"
      - "--slot-save-path"
      - "/slots"
      - "--jinja"
      - "--reasoning"
      - "off"

Plus a warmup automation, without which the first request of the day returns nothing:

- id: local_llm_warmup
  alias: Local LLM - keep prompt cache warm
  triggers:
    - trigger: homeassistant
      event: start
    - trigger: time_pattern
      minutes: /10
  actions:
    - action: conversation.process
      data:
        agent_id: conversation.local_llm_server_gemma4_26b_ai_agent
        text: hello
      continue_on_error: true
  mode: single

And a sidecar that saves and restores the KV slot, covered below.

Results

Median 8.09 s to a complete answer across 5 representative queries: 6.28 s for the time, 9.90 s for tomorrow’s forecast. Committed memory 0.73–1.53 GiB, behind 13.44 GiB of reclaimable page cache. Idle CPU 0.00%.

A voice latency figure means little without saying which moment it measures. A person notices the gap between finishing speaking and hearing the first word; benchmarks report the time to a fully generated response.

2.7 s against 1.0 s is slower, but a fraction of what the end-to-end numbers imply. Getting it out of HA takes care: the tts-start event fires after intent completion, so measuring it makes streaming look switched off. Use tts_start_streaming, inside intent-progress (assist_pipeline/pipeline.py).

How it got there

The measurement that makes CPU inference viable

Eight seconds is a long time to wait for a computer, and it is the first objection to everything above. Almost nothing waits.

prefer_local_intents runs HA’s sentence matcher before the conversation agent: “turn off the kitchen lights” matches a template and executes directly, and the LLM never sees it. Community write-ups quote “about 90% handled locally” as folk knowledge; I wanted my own number. The websocket command conversation/agent/homeassistant/debug matches utterances without executing them, so you can replay months of voice history against a live house without turning on a single light.

Reading the 54 that fell through changed my mind. 38 were false wake-word triggers: the living-room satellite had been catching television dialogue and sending it to a language model in another country.

“Obi Wan and the council don’t trust me.”

Another 10 were commands lacking local coverage. That leaves 6 real knowledge questions in 4 months.

This is what makes the whole approach work. A cloud model has to be fast because it sits on the critical path for everything. A local model that only handles the unusual request can take 8 seconds, because the lights, the timers and the projector never touch it. Eight seconds on a request you make weekly buys a house that keeps working when the broadband stops. The remaining wins live in the matcher: 10 more sentence templates and a wake word that ignores the television. Measure your fallthrough rate before optimising the LLM path.

Everything hard here is about one cached prefix

The prompt has two parts, and they behave completely differently:

Cold, that prefill takes 124 s with repacking, 158 s without. The request outlives the client timeout, so what Home Assistant shows you is an empty response, or the stock “I didn’t understand that, sorry”. Both look exactly like the model failing to answer, so you blame the model. Hence the warmup automation, and the consequence that any edit to your prompt costs one full cold prefill.

Two things then destroy a warm cache that you’d never guess from the flags.

A container restart. --slot-save-path exposes POST /slots/{id}?action=save|restore: 395 ms to save, 353 ms to restore, and after a restart the server prefilled 70 tokens instead of 12,779. Several people have built this properly: an upstream tutorial, stillwarm, and two reverse proxies. Use one of those rather than my 30-line shell loop, which polls /health so it restores whenever llama-server comes back. Steal one detail from all of them: promote a save only above 10,000 tokens, or a save against a cold slot overwrites your good cache file and turns restart protection into a guaranteed cold start.

Extra slots. The same bin query took 132 s once and 7 s another time; one failed outright. It looked like model variance under memory pressure. llama-server defaults to --parallel -1, auto, which here gave four slots sharing one unified KV buffer:

The evidence is a slot-id histogram of 23/15/2/1 and bimodal timings; whether the cause is eviction or simply a request landing on a slot nobody had warmed, I didn’t measure closely enough to say. --parallel 1 fixes it either way.

Two things the README doesn’t mention. The modes divide context differently: under kv_unified = true each slot sees the full n_ctx but shares one buffer, while under kv_unified = false each gets n_ctx / N. That per-slot number is the ceiling llama.cpp measures your prompt against, and it truncates a longer prompt silently: truncated = 1 in the log, and nothing surfaced to HA. And some builds silently override --parallel 1: roughly November to mid-December 2025, -np 1 still produced four slots (#17450, #17989; -np 1 -kvu works around it). Read n_slots from the startup line, not the flag you passed.

RSS counts the model twice

I ran this for a week believing it cost 24.63 GiB. Reading /proc/<pid>/smaps_rollup instead of VmRSS:

Private_Clean is memory the kernel can drop and re-read from disk; Private_Dirty must stay. RSS adds them and calls the total your cost. llama.cpp repacks quantised weights into SIMD-friendly blocks at load, into a fresh anonymous buffer, while the original mmap stays mapped. The model sat in memory twice. Stopping the container confirmed it: system used fell by 14.3 GiB, not 27. bartowski1182 reported this upstream in #12149.

Warm latency doesn’t change. Repacking’s only benefit is cold prefill, and the slot sidecar already reduces that path to ~350 ms, so --no-repack costs 34 seconds exactly once, when a prompt change invalidates the saved slot, and returns 13.5 GiB. (It disables extra buffer types generally, so on an AMX-capable CPU it may cost more than it saves.)

One caveat on the most misreadable number here. “Under 1.5 GiB committed” still needs the 13.44 GiB of page cache behind it; a 26B model does not run in 1.5 GiB. The third arm capped the container at 8 GiB to force demand-paging, and it answered correctly at 2.3× the latency.

Compare traces with a frontier model

“What is the temperature in the office?” was consistently slow. The trace showed GetLiveContext(area='office', domain='sensor') returning “No exposed entities found”, then a retry with another domain, which looks like the small local model fumbling. Claude Haiku makes the same failed lookup and retry. The cause is my house, where that reading lives on a climate entity’s current_temperature attribute, and any agent has to discover that by failing first. A 40-token line in the prompt naming the domain took the query from ~23 s to 8.3 s. A good share of apparent local weakness is architecture, and it follows you to the cloud.

What it costs

Against the six genuine knowledge questions from my logs, Gemma won on four, tied on one, and was far more concise throughout: Haiku emits markdown bullets that Piper reads aloud as “asterisk”. On one it was confidently, fluently wrong, dating the first British Grand Prix to 1954 rather than 1926. Six questions are too few to score the models, but that shape of failure is what to plan for. A long-tail factual error arrives in the same tone as the correct answers.

That’s the trade: a little latency and some factual reliability, bought with a lot of privacy and a house that works offline. For my household it’s a good one, and the rollback is a dropdown. The living-room satellite runs the local model, the kitchen stays on Haiku as a control I can talk to.

What surprised me is how little hardware this needed. Local voice has a reputation for needing a GPU. On a workload where the sentence matcher handles 85% of what it hears, a CPU you already own and 1.5 GiB of committed memory are enough.

Alan Treadway is the founder of Bitlogic Solutions.