What Hardware You Need to Run AI Models Locally

“Can we run AI on our own hardware so data never leaves the company?” Yes, and it is simpler today than two years ago. Before buying, though, you need to know at which hardware tier things stop being simple and start being expensive, and when the numbers favor models available through an API.

Applies to Ollama LM Studio llama.cpp vLLM

Four hardware tiers

On the first tier a model generates a few to a dozen or so words per second. That can be too slow for real-time conversation with a person but is enough for overnight batch processing. OpenAI states that gpt-oss-20b runs in 16 GB of memory, which marks the upper limit of this tier well.¹

A professional card offers three times the memory of a consumer card and detects and corrects memory errors on its own. That starts to matter when the machine runs around the clock and nobody plans to hunt for the cause of results that occasionally come out differently.

The fourth tier holds machines with memory shared by the processor and the graphics chip. The Mac Studio with the M3 Ultra chip, introduced in March 2025, offered up to 512 GB of such memory, but in March 2026 Apple withdrew that option and in May left only 96 GB.² NVIDIA DGX Spark fits 128 GB into a book-sized case.³

What to run them with

In a corporation the license matters in software reviews, which is why LM Studio is the only one of the four that needs separate approval from IT.

How much memory a model takes

A model at full precision takes about 2 bytes per parameter, and compression, or quantization, divides that value at the cost of a small drop in quality.

Machines with shared memory have plenty of it, but processing a long prompt before the first word of the answer takes noticeably longer on them than on a dedicated card. With short queries the difference is invisible; with a hundred-page contract it is obvious.

Power, heat and upkeep

A strong computer with a graphics card needs a power supply of about 1 kW, so under load it draws roughly as much as an electric kettle. A machine running 8 hours a day, 20 days a month uses about 110 kWh. At the EU average electricity price for businesses in the second half of 2025, EUR 18.37 per 100 kWh, that comes to about EUR 20 a month.⁴

The other costs weigh more. Heat released into a small room forces cooling in summer, and the fans under load can be heard all day. Someone has to update drivers, manage model versions and react when the service stops responding on a Friday afternoon. In a smaller company that is usually a person who already has a full set of other duties.

When your own hardware, when an API

The calculation rests on an assumed workstation price:

If the team’s monthly API spending today is lower than that monthly cost, buying hardware will not pay for itself on cost alone.

2026 prices changed the numbers

DRAM prices, the price of working memory, rose by about 90% in the first quarter of 2026 compared with the end of 2025, as manufacturers redirected production to data centers serving AI.⁵ The highest-capacity machine variants are getting more expensive or disappearing from catalogs, and a configuration priced a year ago costs something different today. A calculation that favored your own hardware in 2025 often looks the other way round in 2026, so check the manufacturer’s current price list before deciding to buy.

How to start

1. Hardware the company already has (1 day). Ollama or LM Studio on an existing computer and one first-tier model.
2. One specific process (2 weeks). Measure quality and response time on real tasks, for example ticket classification.
3. Hardware decision (after measuring). A card costing tens of thousands of zloty makes sense only once a model of the given class does the task and the volume justifies a dedicated machine.

Such an experiment costs a few hours of work and settles more than weeks of debating specifications. We set up a local environment with models and first agents as part of the AI Accelerator.

Frequently asked questions