GPU memory calculator for LLMs
Estimate LLM memory requirements: whether a model fits in GPU memory, and what to change when it runs out of memory.
The GPU memory calculator sizes a model for serving on one or more GPUs, and it needs no sign-in or account. If a deploy has already run out of memory, enter the settings that deploy used: the model, **Weight precision**, card, **GPUs**, **Context tokens** and **Simultaneous requests**. Then choose **Calculate** or press **Ctrl+Enter**.
Estimate whether your model and serving workload fit in GPU memory, see memory use for weights, cache and working space, and explore changes that could make it fit. The interactive tool requires a web browser: https://docs.zettabyte.com/docs/zcloud/deploy-and-serve/gpu-memory-calculator
Notes on the inputs
- GPU memory, also called VRAM, is the memory on the GPU card itself, separate from the machine's main memory. The **Memory** field in **Computing Config** is not GPU memory.
- The calculator covers serving a model: it has no input for training or fine-tuning, and it doesn't estimate speed.
- If a model isn't in the list, choose **Custom model…** to enter its details or to paste its config.json. A config with a standard or latent-attention layout is read; a config with state-space or linear-attention layers is refused with a reason. For a card that isn't in the list, choose **Other memory size** and enter its memory in GiB.
How to read the result
The result is an estimate, not a guarantee that the workload will run. For a model the calculator can size, it gives one of three verdicts:
| Verdict | What it means | What to try |
|---|
| Should fit | The estimate fits in the setup's GPU memory | Deploy it and test it on the real run: see [Deploy a model](https://docs.zettabyte.com/docs/zcloud/deploy-and-serve/deploy-a-model) |
| Tight | It fits only at the low end of the working-space range, explained in the **Working space** bullet below the table | Treat it as a risk to test, or change an input to gain a margin |
| Does not fit | The model needs more GPU memory than the setup has | Change the inputs you control, then the card or the GPU count |
A card's reported capacity is usually a little below its labelled size, so a setup at the edge, and a **Tight** result in particular, has little margin left.
The result splits GPU memory per GPU into three parts, each in GiB, and gives the total and the working space as ranges:
- **Weights** are the model's learned values, loaded into GPU memory. They depend on the model's size and the **Weight precision**. At a model's published precision the figure comes from the model's file sizes; at any other precision it is estimated from the parameter count.
- **KV cache** is the memory kept for the text each request is working on. It grows with the context length and with the number of simultaneous requests. For a model with latent attention, each GPU holds the whole cache, so more GPUs do not shrink it.
- **Working space** is the memory the serving software uses while it runs. It's a range because it depends on the serving software, and it widens when **Serving software** is set to **Not sure yet**, so plan for the high end.
A model the calculator cannot size gives a message that says why and shows no figures. This applies to a listed model whose files are not publicly readable, and to a pasted config.json with a layout the calculator does not read. Nothing is guessed.
- For a listed model whose files are not publicly readable, the message begins **Cannot be sized yet**, names the reason and offers no figure to plan on.
- A pasted config.json with a standard or latent-attention layout is read. For a latent-attention layout, the result is labelled **Not a listed model: Estimated**, and the parameter count is not in such a config, so it is entered by hand under **Model details**.
The result does not see everything that uses GPU memory
The result can't see long prompts, other processes on the GPU, software version problems, or settings that multiply memory, such as the number of parallel slots. Compare it with the memory report your serving software prints at start-up. Working space is an estimated range from public engine logs. Not measured on our hardware.
If it does not fit
When the verdict is **Does not fit**, find the largest part of the breakdown and change the input that shrinks it. For example, say a result shows 6 GiB of weights, 44 GiB of KV cache and 2 to 6 GiB of working space against a 40 GiB card. Even with no weights at all, the KV cache and working space would come to 46 to 50 GiB, so a lower precision can't fix it alone; a shorter context or fewer simultaneous requests can.
When the weights are the largest part, or when context and requests are already low, change the inputs you control one at a time, in this order. If **KV cache** is the largest part, start at step 2.
- Lower **Weight precision** to **8-bit** or **4-bit**, or to a model-specific option where one is listed; this changes **Weights** only, and leaves **KV cache** and **Working space** as they were.
- Shorten **Context tokens**; this shrinks **KV cache**, and the weights stay the same. Each request can then hold less text.
- Lower **Simultaneous requests**; this also shrinks **KV cache**, and the weights stay the same. The deploy then serves fewer requests at once.
- Choose a card with more GPU memory, or raise the number of **GPUs**. More GPUs shrink the weights per GPU, not the cache of a model with latent attention, which every GPU holds whole.
Read the result again after each change. After the first **Calculate**, a change to **Weight precision**, **Context tokens** or **Simultaneous requests** updates it in place.
When the result suggests more GPUs, the count is a minimum that assumes the model is split evenly across them, so treat it as a floor and confirm it on the real run.
A **Should fit** or **Does not fit** result also shows the fewest GPUs of the selected card that hold the workload, and says when the number of GPUs selected is higher or lower than that. The count is a minimum under the same even-split assumption, so it is confirmed on a real run.
Next steps
- [Deploy a model](https://docs.zettabyte.com/docs/zcloud/deploy-and-serve/deploy-a-model): once the model fits, deploy it.
- [Call your endpoint](https://docs.zettabyte.com/docs/zcloud/deploy-and-serve/call-your-endpoint): send the deployed model its first request.