Hardware Requirements for an AI Lab

Hardware determines which models fit and whether they run fast enough to use every day. Memory is usually the main constraint. Understanding it before buying anything can save a lot of money and frustration.

The one thing that matters most: memory

When people compare machines for local AI, they often focus on CPU speed, storage, or the name of the GPU. For inference, memory is usually more important. More specifically, the question is how much memory the GPU can access. The model weights need to fit somewhere, and larger models require more of it.

A 7B parameter model in 4-bit quantization needs roughly 4–5 GB of GPU memory just to load. A 13B model needs around 8–9 GB. A 70B model needs more than 40 GB. Those numbers do not include the additional memory used by longer context windows or concurrent requests. Fitting the weights is only the starting point.

Dedicated VRAM vs unified memory

On a desktop or laptop with a discrete GPU, the card has its own dedicated VRAM. It is fast, but fixed. If the GPU has 8 GB of VRAM and the model needs 10 GB, part of the workload must move into system RAM. That may allow the model to load, but performance usually drops significantly.

Apple Silicon uses a unified memory architecture. The CPU and GPU share one pool, so a machine with 64 GB can make most of that capacity available to model inference. This makes large-memory Macs useful for running models that would otherwise require one or more expensive GPUs. Apple charges heavily for larger memory configurations, and the memory cannot be upgraded later.

What about Fedora with a dedicated GPU?

On a Fedora workstation with an AMD or NVIDIA GPU, you are usually working with fixed VRAM. Linux can move some GPU allocations into system memory when VRAM is exhausted. For AMD GPUs, this is managed through the kernel's TTM memory subsystem and the amdgpu driver.

System-memory overflow runs at lower bandwidth than true unified memory. It may help a model load, although performance can fall sharply. NVIDIA supports similar forms of oversubscription through CUDA, with results depending on the workload and software stack.

How to decide how much VRAM you need

Start with the models you actually want to use. For 7B–13B models used for chat or coding assistance, 12–16 GB of VRAM is a practical range. For 30B+ models or fine-tuning, 24 GB becomes a more realistic starting point.

A rough rule for 4-bit quantization is:

model parameters in billions × 0.6 + 2 GB of overhead

This is only an estimate. Architecture, context length, cache size, and inference software all affect actual memory use. Start with what you have, run the models you care about, measure the bottleneck, and upgrade when the workload proves you need it.

Strong alternatives without a discrete GPU

Large-memory systems with integrated graphics can also work well when the accelerator has access to enough fast memory. AMD's Ryzen™ AI Max+ 395, for example, is a credible alternative to a Mac Studio for some local AI workloads.

The same principle still applies:

Start with memory capacity and bandwidth, then verify that your inference stack supports the hardware well.

Recommended starting points

For a Fedora workstation, an NVIDIA RTX 3090 or AMD RX 7900 XTX provides 24 GB of VRAM and is a practical place to start for larger local models. Apple Silicon with a large unified-memory configuration serves a similar purpose. A Beelink GTR9 Pro with a Ryzen AI Max+ 395 and 128 GB of RAM is another option if you want a compact integrated system.

There is no single best machine. The right choice depends on the models you want to run, the software stack you need, and how much speed you are willing to pay for.