Contents
In brief
A Dev.to author reports running the large language model Gemma 4 26B on a Xeon E5 v2 server processor from roughly 2013, with no GPU involved. The reported result is about 12 generated tokens per second while using approximately 45 GB of RAM.
That is not a blueprint for a busy production service. It is, however, a useful reminder that an already-owned server can still support a local proof of concept, infrequent inference, or an internal experiment when its latency budget is generous.
What happened
The original post describes a Linux machine with at least 64 GB of memory, Python 3.10, PyTorch, and Transformers. Its central technique is loading the model at lower precision; the author estimates that this cuts memory use from around 120 GB to about 40 GB.
The example then disables gradient calculation, uses a batch size of one, and applies CPU-oriented optimizations. The post reports a three-to-five-minute cold start, around 150 W of power draw, and roughly 12 tokens per second on a Xeon E5 v2. For comparison, it cites 100–300 tokens per second for modern GPUs.
Those figures should be treated as the author’s measurements, not a general model guarantee. Token rate changes with the weight format, library build, prompt length, CPU topology, memory bandwidth, and even the way input and output tokens are counted.
Why it matters
Local LLM discussions often stop at video memory: without a GPU, the task is assumed to be impossible. In reality, the decision also depends on how quickly an answer is needed. A few seconds may be unacceptable in an interactive product but perfectly workable for a document-processing job, a private lab, or a single-user tool.
Reusing an existing server can also keep data close to the team operating it. That does not remove access-control or operational obligations, but it can avoid transmitting every prompt to an external provider during early evaluation.
The savings are not automatic. A slow startup, continuous power usage, and low concurrency can make a CPU deployment less economical than an API or rented accelerator. The relevant comparison is total cost and observed latency on the workload that will actually run.
In practice
- Budget memory first. The reported setup needs at least 64 GB of RAM and about 200 GB of disk space. Leave headroom for the operating system, model loader, and context.
- Validate the quantization path. Lower-precision loading methods do not all work the same way without a GPU. Reproduce the load on the exact library versions and record the actual weight format.
- Measure a representative workload. Separate cold start, time to first token, generation rate, and peak memory for prompts and responses your users will send.
- Tune the CPU deliberately. Thread count, BLAS implementation, and core affinity can materially change results. Intel MKL may help, but benchmark it against alternatives on the target host.
- Set a clear use boundary. A GPU remains the practical choice for concurrent users or long generations. CPU inference better fits a single user, a test environment, or asynchronous work.
Takeaway
The post does not show that an old Xeon replaces an accelerator: the reported performance gap is substantial, and the implementation details need independent verification. It does illustrate a more useful framing: the feasibility of local AI depends on memory, weight format, latency expectations, and operating cost—not processor age alone.
If a team already owns a high-memory server, a small test on its real prompts is worthwhile. Even a negative result produces the measurements needed to choose a GPU, an API, or a smaller model with confidence.