> See the full index of documentation pages at https://docs.zettabyte.com/llms.txt. # Manifest reference and multi-role jobs > Every spec key, how the four layers merge, and composing one job from several roles. Source: https://docs.zettabyte.com/docs/cli/manifest-reference Last updated: 2026-08-21 ## How a launch is assembled `zb` assembles a launch from up to four layers. Higher layers override lower ones. It validates the merged result as a whole (unknown keys, kind rules, role rules) before it sends anything. | Layer (lowest to highest) | Example | Notes | | --- | --- | --- | | dotenv files | `env_file: .env`, `--secrets-file .env.secrets` | only fill env and secret keys nothing else already set | | spec file | `zb launch train.yaml` | the paved road; `-` reads stdin | | flags | `--gpu h100:8 --env SEED=1` | override the file field by field | | `--set` overrides | `--set serve.replicas.max=3` | win over everything | ## Merge rules - Scalars: a set flag replaces the file's value (`--name`, `--gpu`, `--ttl`, and so on). - `env:` and `secrets:` merge per key. The flag wins on a collision, and other keys from the file survive. - `volumes:` (and every repeatable list flag): the flag list replaces the file's list entirely. Lists never splice. - `--set path=value`: a dotted path into the manifest (`--set env.SEED=42`, `--set serve.replicas.max=3`). The value parses as a YAML scalar, so numbers and booleans keep their types. - `zb` folds dotenv files in last, but they lose to every explicit key. A key in both `env_file` and `secrets_file` with no explicit winner is an error. ## The full manifest The table lists every key the spec accepts. `zb` rejects unknown keys. | Key | Kinds | Meaning | | --- | --- | --- | | `kind` | all | required: `job` \| `dev` \| `serve` \| `box` (or `--kind`) | | `name` | all | auto-generated when omitted; lowercase letters, digits, and dashes, max 40 chars | | `description` | all | free text | | `image` | job, dev, box | any pullable ref, or a preset name like `pytorch`; omit on `job` and `box` for the default preset; serve image mode takes the imported runtime image id | | `run` | job, image-mode serve | the command; required on `job`, rejected on `dev` and `box` | | `workdir` | job, dev, box | ship this local directory, read-only at `/workspace` (1 GiB cap) | | `outputs` | job, dev, box | results mount (absolute container path), downloadable with `zb pull` even after `zb down` | | `outputs_s3` | job, dev, box | write results straight to your own S3 bucket: `{url, region, access_key, secret_key}`; needs `outputs:` set; keys are never stored or shown | | `share` | job, dev, box | mount the tenant share volume at this absolute container path | | `ttl` | all | autoclean after DURATION, anchored at launch | | `resources.gpu` | all | `TYPE[:COUNT]` as listed by `zb gpus`; a bare type means 1; required on `serve` | | `resources.cpu`, `resources.memory` | all | cores / GB; omit on GPU workloads for per-card sizing | | `nodes` | job, serve | job: above 1 = distributed (gang-scheduled), exclusive with `roles:`; serve: shard one replica across N nodes (needs `replicas.min == max`) | | `max_restarts` | job | automatic restart budget; a restart re-runs the whole gang from scratch | | `roles` | job | multi-role gang, see below | | `debug` | job, serve | hold a crashed container instead of restarting it, so logs keep the traceback (see Everyday commands) | | `env` | all | plain env vars | | `secrets` | job, serve | env vars marked sensitive; rejected on `dev` and `box` (see the secrets tiering callout on Launch and specs) | | `env_file`, `secrets_file` | all | local dotenv folded in at lowest precedence | | `volumes` | job, dev, box | `NAME:/mount/path` entries; rejected on `serve` (model artifacts are mounted by the platform) | | `queue` | all | default true: park `QUEUED` on quota exhaustion | | `priority` | job, serve | 1 to 6, higher schedules first | | `serve.model` | serve | managed mode (model name, or a model id starting with `M0`); add `image` plus `run` to serve it on a custom runtime (weights mount at `/home/model`) | | `serve.served_name` | serve | image mode: the model name on the OpenAI API (required in image mode) | | `serve.replicas` | serve | `{min, max}`; default `{1, 1}`; min 0 launches stopped; `zb stop` scales to zero afterwards | | `dev.ide` | dev | `jupyter` (default) \| `vscode`; a workload that runs your image's own ENTRYPOINT is `kind: box` | ## Exact syntax by kind Manifests parse as YAML or JSON. `zb init -k KIND -` prints a ready-to-edit template for any kind. Below is every key at its exact spelling and nesting, per kind. `zb` rejects anything not listed for a kind at validation, before the request goes anywhere. ### kind: job ```yaml kind: job name: llama-ft # optional; lowercase letters, digits, dashes; max 40 description: llama finetune # optional image: pytorch # preset | @TAG | @fav | any pullable ref; omit for the default preset run: torchrun train.py # required resources: # omitted entirely = CPU-only default sizing gpu: h100:8 # TYPE[:COUNT]; a bare type means 1 cpu: 96 # cores (number); omit to size per card memory: 1024 # GB (number); omit to size per card nodes: 2 # >1 = distributed; mutually exclusive with roles: max_restarts: 1 priority: 5 # 1-6, higher schedules first queue: true # false = fail fast when GPU quota is exhausted debug: true # hold a crashed container instead of restarting it workdir: . # local directory to ship (read-only at /workspace) outputs: /outputs # absolute container path # outputs_s3: # optional: results write straight to your own bucket # url: https://ENDPOINT/my-bucket/run-42 # region: us-west-2 # optional # access_key: AKIA... # secret_key: ... # use a bucket-scoped, fine-grained key share: /share # absolute container path ttl: 8h volumes: - training-data:/data # NAME:/mount/path env: WANDB_PROJECT: llama secrets: # job and serve only HF_TOKEN: hf_xxx env_file: .env secrets_file: .env.secrets ``` ### kind: serve ```yaml kind: serve name: qwen-serve resources: gpu: h100:1 # required on serve serve: model: qwen3-8b # managed mode: model name, or a model id starting with M0 replicas: # optional; default {min: 1, max: 1}; min 0 = launch stopped min: 1 max: 3 # custom runtime for a managed model: keep serve.model, add image + run # (weights mount at /home/model): # image: IMG01... # imported runtime image id (zb image import prints it) # run: python3 -m vllm.entrypoints.openai.api_server --model /home/model ... # image-only mode (no serve.model; requires tenant enablement): # image: IMG01... # run: python3 -m vllm.entrypoints.openai.api_server ... # serve: # served_name: my-model # required in image mode: the name on the OpenAI API # nodes: 2 # shard ONE replica across N nodes (model too big # # for one node); needs replicas min == max # also valid: description, env, secrets, env_file, secrets_file, ttl, # queue, priority, debug. volumes: and share: are rejected on serve ``` ### kind: dev ```yaml kind: dev name: notebook-1 image: pytorch # preset | @TAG | any pullable ref dev: ide: vscode # jupyter (default) | vscode # also valid: description, resources, workdir, outputs, share, ttl, # volumes, env, env_file, queue. run: and secrets: are rejected on dev ``` ### kind: box ```yaml kind: box name: dev1 image: pytorch # a preset gets a keep-alive shell; a custom image's ENTRYPOINT must keep running # also valid: description, resources, workdir, outputs, share, ttl, # volumes, env, env_file, queue. run: and secrets: are rejected on box ``` ## Multi-role jobs `roles:` composes one job out of several differently shaped pod groups. Examples: a trainer role beside a rollout role serving vLLM (the disaggregated-RL / verl topology), or a parameter server beside workers. Use it anywhere unlike pods must live and die as one unit. ```yaml kind: job name: rl-disagg image: ghcr.io/acme/rl:cu124 # top-level values are the defaults resources: gpu: h100:4 roles: - name: trainer # each role overrides what it needs run: python train.py - name: rollout run: python -m vllm.entrypoints.openai.api_server --model /ckpt resources: gpu: h100:8 # gpu overrides as a TYPE:COUNT unit replicas: 2 # pods for this role (default 1) env: VLLM_PORT: "8000" # merged over the job-level env ``` - `roles:` is job only, and `nodes:` is mutually exclusive with it. The gang size is the sum of role replicas. - Defaults inherit downward. A role that omits `image`, `run`, or `resources` uses the job's top-level value. Every role must end with an effective image and run. `resources.gpu` inherits as one `TYPE:COUNT` unit. `cpu` and `memory` inherit individually. Role `env` merges over job `env`. - Role names are lowercase letters, digits, and dashes (max 40), unique within the job. The name is what you see in `zb pods` (`trainer-0`, `rollout-1`). It is also the role's network hostname inside the job. - One gang: the platform co-schedules every role as one unit. The job starts only when every pod of every role can start. All roles share the job's network, so intra-job DNS reaches every role across nodes. - Tooling is role-aware. Multi-pod `zb logs` prefixes lines per pod. It narrows with `--role rollout`, or with `--role trainer --replica 1` for one raw stream. `--role` and `--pod` pick the target for `zb shell`, `zb exec`, and `zb cp`. `zb pods NAME` is the canonical list. ## Topology rules for RL post-training RL fine-tuning runs colocated. The rollout engine (vLLM) lives in the same job as the trainer and shares its GPUs (TRL's `vllm_mode` colocate, verl's hybrid engine). The topology rule on this platform is simple. Jobs have no inter-workload networking or ports, so the trainer cannot reach a separate rollout server. A `serve` endpoint cannot receive weight updates either. A frozen reward model or judge is the exception. Serve it with `kind: serve` and call its public URL (`zb url`, `zb key`) from the training loop. A single-pod colocated job does not need `roles:` at all. That is a plain job. ## Distributed and multi-node jobs Set `nodes: N` and the job gang-schedules across nodes. The platform injects `MASTER_ADDR`, `MASTER_PORT`, `WORLD_SIZE`, and `RANK` into job pods. A `torchrun` command that reads those variables runs a distributed trainer with no extra wiring (rank 0 hosts the rendezvous). Request the node's full GPU complement per node (`gpu: h100:8`). Multi-node is meant to run whole nodes. If job pods have no Hugging Face or PyPI egress, bake dependencies and weights into the image. Weights cannot ride the workdir bundle (1 GiB cap). Two gotchas. First, `nodes: N` sizes N pods, not guaranteed N machines. Whole-node GPU sizing is what forces the spread. Second, `RANK` and `WORLD_SIZE` are node-level, not global. `torchrun` and `accelerate` (`--machine_rank`/`--num_machines`) expand them correctly. A raw `torch.distributed` `env://` launch with one process per GPU must recompute the global rank itself (`RANK*nproc + local_rank`). ## Multi-node serve `nodes: N` on a `serve` shards one replica across N nodes (a leader plus N-1 workers). Use it for a model too big for a single node. It needs `replicas.min == max`, because a sharded replica cannot autoscale. Unlike jobs, the platform does not run a launcher for you. Every pod runs the same `run:`. You differentiate the worker yourself off `$LWS_WORKER_INDEX` (0 = leader), with `$LWS_LEADER_ADDRESS` as the head address. The platform injects the cross-node network interface as `NCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME`. ```yaml run: | EXTRA=""; [ "${LWS_WORKER_INDEX:-0}" != "0" ] && EXTRA="--your-frameworks-worker-flag" your-serve-cmd --head-address "$LWS_LEADER_ADDRESS" $EXTRA ... ``` Two things will cost you time if you skip them. First, launch with `--debug`. Otherwise the platform recreates a crashed serve pod before you can read the traceback. Second, raise your serving framework's engine-ready timeout. Big models take many minutes to initialize across nodes, and the default is often too short. The framework's own multi-node environment (host-IP overrides, shared-memory transport, ready timeout) is your responsibility.