I run a local LLM stack across an M1 MacBook Air and a Proxmox host with an AMD Ryzen 7 255 and integrated Radeon 780M graphics. It serves models over an OpenAI-compatible API and a web interface, and it cost nothing beyond hardware I already owned.
It is also slower than an API by an enormous margin, and the GPU I assumed would carry it never got used. Both of those are the interesting parts.
Why bother
Three reasons, in order of how much they mattered to me.
Cost predictability. Not “free”, since the hardware and the electricity are real, but fixed. Iterating on a prompt two hundred times does not produce a bill that scales with my willingness to experiment.
Data that does not leave. Some things I want a model to read are client material or my own infrastructure config. A local model makes that a non-question rather than a policy discussion.
And understanding. Running inference yourself teaches you what a context window costs, what quantisation trades away, and why the same model behaves differently on two machines. That knowledge transfers directly to using hosted APIs well.
The stack
Open WebUI / API clients
│
v
┌────────────────────────┐
│ gateway │ JWT auth · model routing
└───────┬────────┬───────┘
│ │
┌─────┘ └─────┐
v v
Ollama llama.cpp (vLLM evaluated, not adopted)
│ │
└────── models ──────┘
(GGUF)The gateway exists because the runtimes disagree about everything: how models are named, how they are loaded, how streaming is framed, and what happens on cancellation. One stable contract in front of them means the clients do not care which backend answered.
Choosing between the runtimes came down to a short list:
| Runtime | Where it wins | Where it does not |
|---|---|---|
| Ollama | Simplest model management, good API, systemd service | Less control over low-level inference settings |
| llama.cpp | GGUF handling, CPU and mixed hardware, fine-grained flags | More assembly required |
| vLLM | Excellent throughput on supported GPUs | Impractical on unsupported low-resource hardware |
| Open WebUI | The interface people want | Does no inference at all, it is a client |
That last row caused more confusion than it should have. Open WebUI is a
frontend. If it returns nothing, the question is which backend it was pointed
at, not whether the UI is broken. Mine returned ERR_EMPTY_RESPONSE for an
afternoon because the container listens internally on 8080 and I had mapped the
host port to something else. Nothing was wrong with the model, the runtime, or
the UI.
The hard part: the GPU that was not used
The Proxmox host has a Radeon 780M. I expected to hand inference to it. It ran entirely on CPU, and getting it otherwise turned out to depend on a chain where every link has to hold:
whether the GPU architecture is in the supported list at all, whether the kernel and driver expose it, whether the installed ROCm version matches, whether the inference binary was compiled with that acceleration, whether the container can see the device nodes, and whether an integrated APU sharing system memory is a case that runtime handles.
Integrated graphics is where several of those links get thin. Support tends to target discrete accelerators, and an APU has a different memory architecture, so “AMD GPU” being supported does not mean this AMD GPU is.
Rather than chase it, I measured the CPU fallback and decided against it on numbers.
The numbers
Loading a roughly 17 GB 27B-parameter quantised model on the Proxmox host:
model 27B quantised, ~17 GB
context 4096
processing 100% CPU
decode speed 2.89 tokens/second2.89 tokens per second is about 130 words in a minute and a half. For a paragraph of output you wait around a minute. For anything conversational it is unusable, and no prompt engineering makes that number better.
This is the single most useful thing I got out of the project, because it is a number rather than an impression. It sets the boundary precisely: this stack is for work that is batch-shaped and unattended, not for anything a human waits on.
What it is genuinely good for
Batch jobs where latency is irrelevant. Classifying or summarising a queue of documents overnight costs nothing and nobody is watching the cursor.
Anything touching private data, where the alternative is not a faster model but no model.
Development against the API shape. Building and debugging an agent loop, tool calling, or a retrieval pipeline mostly exercises plumbing. Running that against a local endpoint during iteration and switching the base URL for real work is straightforward, since it is OpenAI-compatible on both sides.
Smaller models interactively. The 27B number is the worst case. Models sized to the hardware behave very differently, and the M1 is a different machine entirely for this work.
Lessons
Measure before you architect. I designed a gateway, planned retrieval with a vector store, and thought about per-user quotas before I had a tokens-per- second figure. Thirty minutes of benchmarking would have reordered the whole project.
“Supported” is not a boolean. Hardware acceleration support is a chain of six or seven conditions, and integrated GPUs fail somewhere in the middle of it often enough to plan for. Budget the time to find out, and decide in advance what you do if the answer is no.
A gateway in front of swappable backends earns its keep immediately. Not because I planned to run three runtimes, but because I changed my mind about which one to use twice, and the clients never noticed.
Local and hosted are not competitors. They have different shapes. Local is fixed-cost, private, slow, and always available to me. Hosted is fast, metered, and someone else’s uptime. Most of my useful work uses both, chosen per task rather than per principle.
Technologies: Ollama, llama.cpp, vLLM, Open WebUI, GGUF quantised models, ChromaDB, JWT gateway, OpenAI-compatible endpoints, Docker, Proxmox, Apple Silicon.