Run a training job
Run a training script on GPUs by creating a training task in the console.
Create a training task in the console from the form with **Advanced Settings** off. To adapt an existing model with a guided form, see [Fine-tune a model](https://docs.zettabyte.com/docs/zcloud/tasks/fine-tune-a-model).
Before you begin
- Allocated GPU capacity, or be ready to **Join The Queue**. See [GPU instances](https://docs.zettabyte.com/docs/zcloud/compute/gpu-instances) and [The resource queue](https://docs.zettabyte.com/docs/zcloud/compute/resource-queue).
- A container image with your training code and dependencies, preset or custom. See [Container images](https://docs.zettabyte.com/docs/zcloud/compute/container-images).
- A `PVC` to persist outputs. Without it, your results are deleted when the job ends. See [Storage and PVCs](https://docs.zettabyte.com/docs/zcloud/models-data-storage/storage).
Create a training task
- Open [**Training**](https://console.zettabyte.com/task-training/index) from the console menu, in the **AI Infrastructure Rental** workspace (chosen in **Switch Platform**).
- Select **Create Task**.
- In **Name**, enter a name for the task.
- Set the **Image Directory**: the container image to run.
- Set the **Master Command**: the startup command. The default is `/bin/bash -c sleep inf`; replace it with how you actually start training, for example `python train.py`.
- Set the **Computing Config**: **Resource Name**, **GPU Count(Card)**, **CPU(Core)**, and **Memory(GiB)**.
- Set the **Maximum Number of Nodes**: `1` makes a simple task, and a number greater than `1` makes a PyTorch task.
- In **Storage**, select **+ Storage Volume** or **+ S3** and enter the container path.
- Select **Review + Create**.
- On the review screen, select **Create Task**.
Optionally, to set environment variables, turn on **Advanced Settings**: the **Environment Configuration** step has **Environment Variables**.
Storage: keep your outputs
Attach at least one `Storage Volume` (`PVC`) and mount it at a directory in the container. Write checkpoints and weights there so they survive job completion. Optionally add an `Ephemeral Volume` (memory-backed scratch, in `GiB`) and/or `S3`. Mount directories must be unique.
Execution frameworks
The default form sets the type from the node count (step 7). With **Advanced Settings** on, the **Computing Config** step offers these frameworks:
- `Simple`: a single-node job (one worker). Start here for most single-machine training.
- `PyTorch` or `MPI`: multi-node, distributed jobs. Each has a master role and worker roles, up to six roles in all.
If the job runs on one node
Use `Simple`. Choose `PyTorch` or `MPI` only for a distributed job across several nodes.
Priority and retries
These settings are in the **Advanced Settings** wizard.
- `Priority` (1 to 6): a higher value ranks higher. See [How zCLOUD works](https://docs.zettabyte.com/docs/zcloud/introduction/how-zcloud-works).
- `Reclaim Policy` and `Auto Retry` are switches, both off by default.
Monitor the job
On the **Current Tasks** tab of **Training**, open the task to see its details and the tabs **Overview**, **GPU Monitor**, **Logs**, **Events** and **Timeline**.
Save as a template
Turn a working configuration into a reusable `Template` so you do not re-enter everything next time. See [Task templates](https://docs.zettabyte.com/docs/zcloud/tasks/task-templates).
Troubleshooting
- Outputs gone after the job finished: you did not write to a `PVC`; everything outside persistent storage is cleared on completion.
- **Review + Create** does not advance and the form shows `GPU resources are insufficient, please check.`: the requested GPU count is not free. Lower `GPU Count(Card)` or wait for capacity.
- A task with the `PyTorch` or `MPI` framework fails immediately: see [Distributed task fails immediately](https://docs.zettabyte.com/docs/zcloud/troubleshooting/distributed-task-fails-immediately).
Next steps
- [Task templates](https://docs.zettabyte.com/docs/zcloud/tasks/task-templates): save this configuration and reuse it.
- [Fine-tune a model](https://docs.zettabyte.com/docs/zcloud/tasks/fine-tune-a-model): the guided flow, when you do not need a custom command.