> See the full index of documentation pages at https://docs.zettabyte.com/llms.txt.

# Run a training job

> Run a training script on GPUs, from image to saved weights.

Source: https://docs.zettabyte.com/docs/run-on-gpus/run-a-training-job

Last updated: 2026-07-21

## Before you begin

- Allocated GPU capacity, or be ready to **Join The Queue**. See GPU instances and The resource queue.
- A container image with your training code and dependencies, preset or custom. See Container images.
- A `PVC` to persist outputs. Without it, your results are deleted when the job ends. See Storage and PVCs.

## Create a training task

1. Open **Tasks** and use **Create Task** then **Training** (or **Create** on the **Training** tab).
2. Name the task.
3. Set the `Computing Config`: `Resource Name`, `GPU Count`, `CPU`, and `Memory`.
4. Set the `Image Directory`: the container image to run (see Container images).
5. Set the `Master Command`: the startup command. The default is `/bin/bash -c sleep inf`; replace it with how you actually start training, for example `python train.py`.
6. Set the `Maximum Number of Nodes`.
7. Under **Advanced Settings**, attach `Storage` and set environment variables.
8. **Review and Create**.

## Storage: keep your outputs

Attach at least one `Storage Volume` (`PVC`) and mount it at a working path (the convention is `/workspace`); write checkpoints and weights there so they survive job completion. Optionally add an `Ephemeral Volume` (memory-backed scratch, in `GiB`) and/or `S3`. Mount directories must be unique. See Storage and PVCs.

## Execution frameworks

When configuring resources (directly or via a template) you choose a framework:

- `Simple`: a single-node job (one worker). Start here for most single-machine training.
- `PyTorch` or `MPI`: multi-node, distributed jobs. Requires multi-node capacity; `MPI` adds a master role and allows up to six roles.
- `Deployment`: launches a long-running Kubernetes `Deployment`, for always-on workloads rather than a finite job.

> [!NOTE]
> **Use Simple unless you are doing distributed training**
> Choose `PyTorch` or `MPI` only when you actually have multi-node resources and a distributed job. A distributed framework without multi-node capacity will fail to start.

## Priority and retries

- `Priority` (1 to 6): how the scheduler ranks your job against others. See How zCLOUD works.
- `Reclaim Policy` and `Auto Retry`: control whether a failed job is retried or restarted.

## Monitor the job

Open the task to see its `Job ID`, entry `Master Command`, resource usage and node placement, a console into the pod, logs, events, and resource monitoring metrics.

## Save as a template

Turn a working configuration into a reusable `Template` so you do not re-enter everything next time. See Task templates.

## Troubleshooting

- Outputs gone after the job finished: you did not write to a `PVC`; everything outside persistent storage is cleared on completion. See Storage and PVCs.
- Job will not start or stays queued: insufficient capacity or low priority. Check the queue and priority. See The resource queue.
- Distributed job fails immediately: framework set to `PyTorch` or `MPI` without multi-node resources. Use `Simple` or allocate more nodes.