> See the full index of documentation pages at https://docs.zettabyte.com/llms.txt.

# GPU memory calculator for LLMs

> Estimate LLM memory requirements: whether a model fits in GPU memory, and what to change when it runs out of memory.

Source: https://docs.zettabyte.com/docs/deploy-and-serve/gpu-memory-calculator

Last updated: 2026-10-17

The GPU memory calculator sizes a model for serving on one or more GPUs, and it needs no sign-in or account. If a deploy has already run out of memory, enter the settings that deploy used: the model, **Weight precision**, card, **GPUs**, **Context tokens** and **Simultaneous requests**. Then choose **Calculate** or press **Ctrl+Enter**.

Estimate whether your model and serving workload fit in GPU memory, see memory use for weights, cache and working space, and explore changes that could make it fit. The interactive tool requires a web browser: https://docs.zettabyte.com/docs/deploy-and-serve/gpu-memory-calculator

## Notes on the inputs

- GPU memory, also called VRAM, is the memory on the GPU card itself, separate from the machine's main memory. The **Memory** field in **Computing Config** is not GPU memory.
- The calculator covers serving a model: it has no input for training or fine-tuning, and it doesn't estimate speed.
- If your model isn't in the list, choose **Custom model…** to enter it. For a card that isn't in the list, choose **Other memory size** and enter its memory in GiB.

## How to read the result

The result is an estimate, not a guarantee that the workload will run. For a model the calculator can size, it gives one of three verdicts:

| Verdict | What it means | What to try |
| --- | --- | --- |
| Should fit | The estimate fits in the setup's GPU memory | Deploy it and test it on the real run: see [Deploy a model](https://docs.zettabyte.com/docs/deploy-and-serve/deploy-a-model) |
| Tight | It fits only at the low end of the working-space range, explained under the table | Treat it as a risk to test, or change an input to gain a margin |
| Does not fit | The model needs more GPU memory than the setup has | Change the inputs you control, then the card or the GPU count |

A card's reported capacity is usually a little below its labelled size, so a setup at the edge, and a **Tight** result in particular, has little margin left.

The result splits GPU memory per GPU into three parts, each in GiB, and gives the total and the working space as ranges:

- **Weights** are the model's learned values, loaded into GPU memory. They depend on the model's size and the **Weight precision**.
- **KV cache** is the memory kept for the text each request is working on. It grows with the context length and with the number of simultaneous requests.
- **Working space** is the memory the serving software uses while it runs. It's a range because it depends on the serving software, and it widens when **Serving software** is set to **Not sure yet**, so plan for the high end.

For a model the calculator can't size in full, the result is a floor: **At least** a figure in GiB, which the model's weight files need per GPU as published. A quantised version of the model, which is the same model at lower precision, needs less.

- If the floor is above the card's memory, the verdict is **Does not fit**. Try a quantised version of the model, or another model.
- If the floor is below the card's memory, the result may show no verdict, because a floor isn't a fit.

> [!NOTE]
> **The result does not see everything that uses GPU memory**
> The result can't see long prompts, other processes on the GPU, software version problems, or settings that multiply memory, such as the number of parallel slots. Compare it with the memory report your serving software prints at start-up.

## If it does not fit

When the verdict is **Does not fit**, find the largest part of the breakdown and change the input that shrinks it. For example, say a result shows 6 GiB of weights, 44 GiB of KV cache and 2 to 6 GiB of working space against a 40 GiB card. Even with no weights at all, the KV cache and working space would come to 46 to 50 GiB, so a lower precision can't fix it alone; a shorter context or fewer simultaneous requests can.

When the weights are the largest part, or when context and requests are already low, change the inputs you control one at a time, in this order. If **KV cache** is the largest part, start at step 2.

1. Lower **Weight precision** to **8-bit** or **4-bit**, or to a model-specific option where one is listed; this changes **Weights** only, and leaves **KV cache** and **Working space** as they were.
2. Shorten **Context tokens**; this shrinks **KV cache**, and the weights stay the same. Each request can then hold less text.
3. Lower **Simultaneous requests**; this also shrinks **KV cache**, and the weights stay the same. The deploy then serves fewer requests at once.
4. Choose a card with more GPU memory, or raise the number of **GPUs**.

Read the result again after each change. After the first **Calculate**, a change to **Weight precision**, **Context tokens** or **Simultaneous requests** updates it in place.

When the result suggests more GPUs, the count is a minimum that assumes the model is split evenly across them, so treat it as a floor and confirm it on the real run.

## Next steps

- [Deploy a model](https://docs.zettabyte.com/docs/deploy-and-serve/deploy-a-model): once the model fits, deploy it.
- [Call your endpoint](https://docs.zettabyte.com/docs/deploy-and-serve/call-your-endpoint): send the deployed model its first request.