Running Models Locally

The question about running models locally comes up in almost every AI delivery conversation, usually phrased as “can we keep this in-house”. The answer is yes, and it is easier than it was two years ago. The useful part is knowing which hardware tier still counts as simple and where the expensive territory starts.

Four hardware tiers

Tier	Memory	What fits	What it is enough for
Laptop or desktop, no card	16-32 GB RAM	quantised models of 4-14 billion parameters	classification, text rewriting, simple extraction
Desktop with a graphics card	24-32 GB card memory	30B models lightly compressed, 70B heavily	coding assistant, summarising, knowledge-base search
Workstation, professional card	96 GB with error correction	the same models, but running continuously	round-the-clock work without chasing results that drift
Memory shared by processor and card	128-512 GB	models that would otherwise need several cards	experiments with the largest open models

An ordinary laptop or desktop, 16 to 32 GB of memory. With no dedicated graphics card, this tier holds the quantised models from the first row of the table. They work, they produce a handful to a couple of dozen tokens per second on CPU, and they handle classification, text rewriting and simple field extraction from documents. OpenAI states that gpt-oss-20b runs within 16 GB of memory, which marks the ceiling of this tier fairly well. That throughput is often too slow for real-time conversation with a person and entirely sufficient for overnight batch processing.

A machine with a 24 to 32 GB graphics card. A high-end consumer card offers 32 GB of its own memory and is the first tier where 30B-class models fit under light compression and 70B models under heavy compression. This is where genuinely useful work begins: coding assistance, summarising long documents, retrieval over an internal knowledge base.

A workstation with a professional card. A professional card carries three times the memory of a consumer one and adds a mechanism that detects and corrects memory glitches on its own. The difference is not only capacity: that correction and vendor support start to matter once the machine runs around the clock and nobody wants to hunt down why a result occasionally comes out different.

Machines with large unified memory. The Mac Studio with the M3 Ultra chip, introduced in March 2025, offered up to 512 GB of memory shared between the processor and the GPU (graphics processing unit), which meant it could hold models that would otherwise require several graphics cards. NVIDIA’s DGX Spark applies the same idea on NVIDIA hardware, with 128 GB of coherent memory in a box the size of a hardback book.

What to run it with

The software layer is now simpler than the hardware. Most of the ecosystem grew on llama.cpp and the ggml library, with GGUF, the file format used to publish compressed models. Ollama reduces model startup to a single command and exposes an OpenAI-compatible API (application programming interface), so swapping providers inside an existing application often comes down to changing an endpoint. LM Studio offers comparable capability through a graphical interface with a built-in model browser, and suits people who do not work in a terminal. For serving several concurrent users, vLLM is the right tool, since it uses the card far more efficiently under parallel load. One licensing note that matters in a corporate context: llama.cpp, Ollama and vLLM are open, while LM Studio remains closed source.

How much memory a model needs

The rule of thumb good enough for planning rests on a single conversion: a model at full accuracy takes roughly two bytes per parameter, and compression divides that figure at a small cost in quality.

Variant	Bytes per parameter	30 billion parameter model
full accuracy	approx. 2	approx. 60 GB
eight-bit	approx. 1	approx. 30 GB
four-bit	approx. 0.5	approx. 15 GB

On top of the figure from the table comes the context cache, which grows with prompt length. This is the source of the most common planning mistake: the model fits the card on paper and stops fitting once several tens of thousands of context tokens are loaded.

The second surprise concerns unified-memory machines. They hold plenty of memory, but processing a long prompt before the first output token takes noticeably longer on them than on a dedicated card. With short queries the difference is invisible. With a hundred-page contract it is impossible to miss.

Power, heat and maintenance

A high-end graphics card draws roughly 575 W under full load, the professional version 600 W, and the whole machine wants a power supply in the 1000 W class, about what an electric kettle pulls.

A machine running eight hours a day across twenty working days consumes around 110 kWh. At the EU business average of EUR 18.37 per 100 kWh that comes to about EUR 20 a month.¹ Electricity rarely decides the outcome.

The rest carries more weight. Six hundred watts released into a small room behaves like a heater, raising summer temperatures by several degrees and forcing a cooling decision. The fans under sustained load are audible all day. Someone has to update drivers, track model versions and respond when the service stops answering on a Friday afternoon. In a smaller company that someone usually already has a full set of other responsibilities, and that is a cost no hardware price list shows.

Where the line against API sits

Three situations where local inference holds up without stretching the argument:

• Data that cannot leave the organisation’s own infrastructure for contractual or regulatory reasons, with the caveat that a provider’s commitments on processing region and zero retention are frequently sufficient, and worth checking before buying hardware.
• High and steady volume of simple tasks: classification, field extraction from documents, embedding generation. Unit cost under constant load comes out clearly lower on local hardware.
• Work without connectivity, or on a network isolated from the internet.

In most other cases the arithmetic favours API. Assume a workstation costing 25 000 to 40 000 PLN amortised over three years: that comes to 700 to 1 100 PLN a month before anyone touches it, plus electricity, plus administrative time. If a team currently spends under roughly a thousand PLN a month on API, buying hardware will not pay back on cost alone, and there is a quality gap to accept as well. The best open-weight models are good, yet on tasks that demand extended reasoning they still trail the frontier models available through API.

2026 has moved the numbers

The calculation is worth doing on current prices, because they have shifted sharply. Prices of DRAM (dynamic random-access memory) rose roughly 90% in the first quarter of 2026 compared with the end of 2025², as manufacturers redirected production toward AI data centres. The effects show up in the price lists of large unified-memory machines: the highest-capacity variants either cost more or leave the line-up, and a configuration priced a year ago costs something different today. A case that favoured owned hardware in 2025 often points the other way in 2026, so check the vendor’s current price list before deciding, not a note from last quarter.

A practical way in

Start at the cheapest tier every time. Ollama or LM Studio on hardware the company already owns, one concrete process, two weeks of measuring output quality and response time. That experiment costs a few hours of work and settles more than weeks of specification debate. Only once it is clear that a model of that class actually does the job, and that volume justifies a dedicated machine, does a conversation about a several-thousand-euro card make sense.

This piece reflects practical research into hardware and cost questions.

Footnotes

1. Eurostat, electricity prices for non-household consumers, second half of 2025: https://ec.europa.eu/eurostat/web/products-eurostat-news/w/ddn-20260508-2. ↩

2. Counterpoint Research, memory prices up to 90% higher than in the fourth quarter of 2025: https://counterpointresearch.com/en/insights/Memory-Prices-Surge-Up-to-90-From-Q4-2025. ↩