Hardware Requirements for an AI Lab

Hardware determines which models fit and whether they run fast enough to use every day. For local AI, memory is usually the main constraint.

Memory matters most

The first question is how much memory the accelerator can access.

A 7B model in 4-bit quantization needs roughly 4–5 GB just for the weights. A 13B model needs around 8–9 GB. A 70B model needs more than 40 GB. Longer context windows, KV cache, and concurrent requests increase that further.

Fitting the model is only the starting point.

Dedicated VRAM vs unified memory

Discrete GPUs have fixed VRAM. If the model exceeds it, part of the workload may spill into system memory, usually with a significant performance penalty.

Unified-memory systems allow the CPU and GPU to share a larger memory pool, making it possible to run larger models without multiple GPUs. The tradeoff is that high-memory configurations can be expensive and are often not upgradeable.

How much memory do you need?

For 7B–13B models used for chat or coding, 12–16 GB is a practical range.

For 30B-class models, 24 GB is closer to the minimum if you want most or all of the model on the accelerator.

A rough estimate for 4-bit inference is:

model parameters in billions × 0.6 + 2–4 GB of overhead

Treat this as a floor, not a buying recommendation. Context length, model architecture, quantization, inference engine, and concurrency all affect real memory use.

Fine-tuning is a different workload and can require considerably more memory.

Start with what you have, run the models/families you care about, research and estimate their memory requirements, then buy enough headroom for context and speed.