November 10, 2026 · 8 min read
A 30B coding model on a CPU mini-PC. 17 tokens a second, and the prompt is the limit.
The question was simple: can the mini-PC that already runs my agent harness also run a coding model? No GPU, no new hardware, zero cost per call, and nothing leaves the house.
The answer is yes, with one catch that matters more than the headline number. This post is about that catch.
The box
- Intel Core i9-12900HK, 32 GB DDR4-3200, no GPU.
- It already runs the harness, LiteLLM, a chat UI and the AgenticTube video renders. The model has to share.
First try: stream a huge model from disk
I started with colibri, a project that runs very large MoE models (284B and up) by streaming experts from NVMe instead of holding them in RAM. It works. DeepSeek V4 Flash ran at about 0.9 tokens a second, even from fast NVMe. That is a demo, not a tool, so I dropped it.
The cloud version of that idea is not better. Streaming experts needs instance-store NVMe (EBS at about 1 GB/s is too slow), which means large memory or storage instances at roughly 2 to 5 USD an hour, so 1.5k to 4k USD a month for 1 to 6 tokens a second. A hosted API is far cheaper for the same models.
Why a 30B model is the fast option
On a CPU, generating a token means reading the weights that token uses from RAM. Memory bandwidth is the ceiling, so speed follows active parameters, not total size.
Qwen3-Coder-30B-A3B is an MoE model with about 3B parameters active per token. At 4-bit, that is roughly 2 GB read per token, which puts the theoretical ceiling for dual-channel DDR4 somewhere in the mid-20s tokens a second. A dense 14B coder would read about four times as much per token and be about four times slower, even though it is “smaller”. There is no smaller Qwen3-Coder anyway: Ollama ships 30B-A3B and 480B.
So I picked qwen3-coder:30b-a3b (Q4_K_M) and, as the alternative, Gemma 4 26B-A4B (QAT).
How it runs
- Ollama on loopback only, one model loaded and one request at a time, 16K context, capped at 24 GB of memory and a low CPU weight, so the harness and the renders win when they compete.
- LiteLLM exposes two aliases,
local-codeandlocal-code-alt, priced at zero with a long timeout and a fallback from one to the other. Apps never talk to Ollama directly. - An on/off switch.
sudo local-llm offstops Ollama, and deploys respect it. With it off,local-codefails at once instead of hanging, which matters more than it sounds when a scheduled job is waiting on it.
The LiteLLM side is a few lines:
model_list:
- model_name: local-code
litellm_params:
model: ollama_chat/qwen3-coder:30b-a3b-q4_K_M
api_base: http://<ollama-host>:11434
timeout: 900
model_info:
input_cost_per_token: 0
output_cost_per_token: 0
Putting it behind the same gateway as the cloud models means every app gets it through the same key and budget rules, and swapping it for a hosted model later is a config change.
The numbers
| Model | Writing | Reading the prompt | Cold load |
|---|---|---|---|
| Qwen3-Coder 30B-A3B, Q4_K_M | 16.8 tok/s | 65 tok/s | 17 s |
| Gemma 4 26B-A4B, QAT | 13.8 tok/s | 59 tok/s | 14 s |
Seventeen tokens a second of output reads fine. You can follow it as it writes.
One oddity: ask Qwen what it is and it sometimes says “Claude”. That is a training-data artifact. The gateway logs say which model actually answered, which is one more reason to put it behind one.
The catch: the prompt
Generation is bound by memory bandwidth. Reading the prompt is bound by compute, the matrix work a GPU is built for, and a laptop CPU does it at about 60 tokens a second.
That is fine for a question. It is not fine for a coding agent, which sends the whole context on every turn: system prompt, tool definitions, open files, the conversation so far. A 10K-token context takes about two and a half minutes before the first output token. Output also slows as context grows: about 15 tokens a second after a short prompt, about 9 after a 2.4K-token one.
Even the chat UI feels it. With its house-style prompt and one tool schema, around 1K tokens before I type anything, the first word takes 15 to 20 seconds, plus 16 seconds of cold load after five idle minutes.
Tuning that did not help
I tested the usual levers on Ollama 0.31.2 with a 2,448-token prompt and 300-token answers:
| Setup | Short: write | Long: read | Long: write | Long total |
|---|---|---|---|---|
| 6 threads, f16 KV cache (default) | 16.9 | 61.7 | 8.9 | ~73 s |
| 10 threads | 18.2 | 44.9 | 10.4 | ~83 s |
| 14 threads | 18.8 | 51.1 | 11.7 | ~73 s |
| q8_0 KV cache, 6 threads | 17.0 | 32.5 | 10.1 | ~105 s |
| q8_0 KV cache, 10 threads | 18.4 | 30.9 | 11.7 | ~105 s |
- Flash attention was already on. Ollama’s
autosetting enables it on this CPU. - A quantized KV cache is a loss on CPU. q8_0 saves about 0.7 GB but halves prompt reading, because every attention step pays to dequantize it.
- More threads help short answers by about 2 tokens a second and tie on long requests, while taking every core away from everything else. These were single runs with about 15% noise. Ollama picks 6, the performance cores, and I kept that.
- Plain llama.cpp instead of Ollama would gain little. Ollama already runs it.
I kept every default.
What would actually help
- Smaller prompts. Retrieval does not make generation faster, but it keeps the context small, and the context is the bottleneck. The box already runs an embedding model for my notes search.
- A GPU, or Apple silicon. My M2 Pro Mac has about 200 GB/s of memory bandwidth and a GPU for the prompt. My estimate, not a measurement, is 40 to 50 tokens a second of output and several hundred of prompt reading. The cost is that it sleeps and travels, so the gateway would try the Mac first and fall back to the mini-PC.
- More RAM, same speed class. A 2×32 GB upgrade, about 100 to 150 EUR, would fit Qwen3-Coder-Next 80B-A3B at 4-bit. It has the same active size, so roughly the same speed with a smarter model.
Verdict: usable, not good
About 17 tokens a second is roughly three times human reading speed. That is fine for chat and single-file edits, and low for agentic coding.
Some reference points for dense models at Q4, in tokens a second of output. These are typical ranges, not my measurements, except the last row:
| Setup | 7-8B Q4 | 30-32B Q4 | 70B Q4 |
|---|---|---|---|
| RTX 4090 / 3090 | 90-130 | 35-50 | 18-25 |
| M2/M3 Max | 50-70 | 18-25 | 8-12 |
| CPU only (DDR5) | 5-10 | 2-4 | 1-2 |
| This box, MoE 30B-A3B (measured) | 17 |
The last row is why the MoE choice matters. A dense 30B on this CPU would sit in the low single digits. With only about 3B parameters read per token, this model runs at small-model speed with 30B-class quality.
What it costs in practice, at about 17 tokens a second:
- A 500-token answer: about 30 s. Tolerable.
- A 2,000-token refactor or file rewrite: about 2 minutes, more as the context grows. You notice.
- An agent loop with 5 tool calls that each emit 300 tokens: about 90 s of pure decode, plus reading the whole prompt again on every turn.
Things that matter more than raw output speed for coding agents:
- Prefill speed. Long context (repo files, tool results) is where the time goes. Watch time to first token, not tokens a second. A 30K-token prompt at 500 tok/s of prefill, a good GPU number, is already 60 s before the first token. At this box’s 60 tok/s it would take over 8 minutes, and it does not even fit the 16K context I configured.
- Parallel requests. If your agent fires subagents or you run several chats, total throughput matters more than single-stream speed. This setup serves one request at a time on purpose, so a second caller waits.
- The quantization trade-off. Dropping from Q4 to Q3, or to a smaller model, to gain speed usually hurts code quality more than the speedup helps. I stayed at Q4.
If your 20 tok/s comes from a dense 30B-class model at Q4 on a single consumer GPU or a Max chip, that is normal and I would keep it. If it comes from a 7B model, something is wrong: check for the KV cache offloaded to the CPU, missing flash attention, or Ollama running on the CPU instead of the GPU (ollama ps shows the split).
For me it stays a batch worker: summaries, first drafts and background jobs that can wait a few minutes and should not cost per call or leave the house. The interactive agent loop stays on hosted models until the prompt can be read faster.