Will it run? Models by GPU memory
A quick reference for whether a model will load on a given amount of GPU memory. Use it to rule options in or out, then check the details for your exact model.
Approximate memory to load a model
These figures are for the model weights only. Allow extra for the context you send and for the software itself.
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| About 7 to 8 billion parameters | about 16GB | about 8GB | about 5GB |
| About 13 to 14 billion | about 28GB | about 14GB | about 8GB |
| About 30 to 34 billion | about 65GB | about 33GB | about 19GB |
| About 70 billion | about 140GB | about 70GB | about 40GB |
Memory on the hardware we sell
| Hardware | Memory |
|---|---|
| RTX PRO 2000 | 16GB |
| RTX PRO 4000, L4 | 24GB |
| RTX PRO 4500 | 32GB |
| RTX PRO 5000, L40 | 48GB |
| RTX PRO 5000 72GB | 72GB |
| RTX PRO 6000 (all editions) | 96GB |
| DGX Spark | 128GB unified memory |
How to read the two tables together
- Pick the row for your model size and the column for the precision you plan to use.
- Find hardware with comfortably more memory than that figure.
- If you plan to fine-tune and not just run the model, you need more again. See Which GPU do I need for fine-tuning?
Lower precision saves memory at some cost to output quality. How much depends on the model, so test before you rely on it.