> See the full index of documentation pages at https://docs.zettabyte.com/llms.txt.

# Run a training job

> Run a training script on GPUs by creating a training task in the console.

Source: https://docs.zettabyte.com/docs/zcloud/tasks/run-a-training-job

Last updated: 2026-11-05

Create a training task in the console from the form with **Advanced Settings** off. To adapt an existing model with a guided form, see [Fine-tune a model](https://docs.zettabyte.com/docs/zcloud/tasks/fine-tune-a-model).

## Before you begin

- Allocated GPU capacity, or be ready to **Join The Queue**. See [GPU instances](https://docs.zettabyte.com/docs/zcloud/compute/gpu-instances) and [The resource queue](https://docs.zettabyte.com/docs/zcloud/compute/resource-queue).
- A container image with your training code and dependencies, preset or custom. See [Container images](https://docs.zettabyte.com/docs/zcloud/compute/container-images).
- A `PVC` to persist outputs. Without it, your results are deleted when the job ends. See [Storage and PVCs](https://docs.zettabyte.com/docs/zcloud/models-data-storage/storage).

## Create a training task

1. Open [**Training**](https://console.zettabyte.com/task-training/index) from the console menu, in the **AI Infrastructure Rental** workspace (chosen in **Switch Platform**).
2. Select **Create Task**.
3. In **Name**, enter a name for the task.
4. Set the **Image Directory**: the container image to run.
5. Set the **Master Command**: the startup command. The default is `/bin/bash -c sleep inf`; replace it with how you actually start training, for example `python train.py`.
6. Set the **Computing Config**: **Resource Name**, **GPU Count(Card)**, **CPU(Core)**, and **Memory(GiB)**.
7. Set the **Maximum Number of Nodes**: `1` makes a simple task, and a number greater than `1` makes a PyTorch task.
8. In **Storage**, select **+ Storage Volume** or **+ S3** and enter the container path.
9. Select **Review + Create**.
10. On the review screen, select **Create Task**.

Optionally, to set environment variables, turn on **Advanced Settings**: the **Environment Configuration** step has **Environment Variables**.

## Storage: keep your outputs

Attach at least one `Storage Volume` (`PVC`) and mount it at a directory in the container. Write checkpoints and weights there so they survive job completion. Optionally add an `Ephemeral Volume` (memory-backed scratch, in `GiB`) and/or `S3`. Mount directories must be unique.

## Execution frameworks

The default form sets the type from the node count (step 7). With **Advanced Settings** on, the **Computing Config** step offers these frameworks:

- `Simple`: a single-node job (one worker). Start here for most single-machine training.
- `PyTorch` or `MPI`: multi-node, distributed jobs. Each has a master role and worker roles, up to six roles in all.

> [!NOTE]
> **If the job runs on one node**
> Use `Simple`. Choose `PyTorch` or `MPI` only for a distributed job across several nodes.

## Priority and retries

These settings are in the **Advanced Settings** wizard.

- `Priority` (1 to 6): a higher value ranks higher. See [How zCLOUD works](https://docs.zettabyte.com/docs/zcloud/introduction/how-zcloud-works).
- `Reclaim Policy` and `Auto Retry` are switches, both off by default.

## Monitor the job

On the **Current Tasks** tab of **Training**, open the task to see its details and the tabs **Overview**, **GPU Monitor**, **Logs**, **Events** and **Timeline**.

## Save as a template

Turn a working configuration into a reusable `Template` so you do not re-enter everything next time. See [Task templates](https://docs.zettabyte.com/docs/zcloud/tasks/task-templates).

## Troubleshooting

- Outputs gone after the job finished: you did not write to a `PVC`; everything outside persistent storage is cleared on completion.
- **Review + Create** does not advance and the form shows `GPU resources are insufficient, please check.`: the requested GPU count is not free. Lower `GPU Count(Card)` or wait for capacity.
- A task with the `PyTorch` or `MPI` framework fails immediately: see [Distributed task fails immediately](https://docs.zettabyte.com/docs/zcloud/troubleshooting/distributed-task-fails-immediately).

## Next steps

- [Task templates](https://docs.zettabyte.com/docs/zcloud/tasks/task-templates): save this configuration and reuse it.
- [Fine-tune a model](https://docs.zettabyte.com/docs/zcloud/tasks/fine-tune-a-model): the guided flow, when you do not need a custom command.
