# Jobs and logs A job is a folder (the bundle), a command, what machine it needs, and its env and secrets. When its machine fails under it, the job restarts on another machine ([Restarts](#restarts)); otherwise it runs once, and a retry is a new job. Job ids look like `j-4k2m9x7qpz`. ## What gets packed `o41-rrun run` packs the current folder, or the one `-C ` names: - In a git work tree: the tracked files, plus untracked files `.gitignore` does not exclude. What git would commit, uncommitted edits included. - Anywhere else: everything except `.git`. Symlinks stay symlinks and the executable bit is kept; submodules and sockets are skipped with a note. A bundle is at most 95 MiB gzipped, and a folder over 2 GiB is refused before packing. Leave data, checkpoints and environments out with `.gitignore` and fetch them inside the job. Bundles are stored by their sha256, so running the same folder again uploads nothing. ## The command Everything after the flags is the command. Flags end at the first argument that is not a flag, or at `--`: ```bash o41-rrun run --cpu-only python train.py --lr 3 # --lr 3 goes to train.py o41-rrun run --gpu L4 -- 'python prep.py && ls out' # one argument: a shell script ``` One argument runs as a shell script as given. Several are quoted into one command line. Either way it runs under `sh -c` in a fresh directory holding the unpacked bundle, in its own process group. The command is at most 16 KiB. ## Env and secrets ```bash o41-rrun run --gpu A6000 --env DATASET=small --env WANDB_MODE --secret HF_TOKEN -- ./train.sh ``` - `--env K=V` sets a variable. `--env K` copies `K` from your shell and fails when it is not set. Repeatable. - `--secret` works the same way, but the value is sealed at rest, never shown back (`status` lists only names) and deleted when the job ends. - Names are letters, digits and `_`, not starting with a digit. Env and secrets together are at most 64 KiB. The command also gets `RRUN_JOB_ID` and `RRUN_ATTEMPT` ([Restarts](#restarts)). The machine's own rrun token is never in the command's environment. ## Images On the container clouds (Vast and RunPod), `--image ` picks the container image. Without it, rrun uses the cloud's Ubuntu image with CUDA that the host's driver supports. Verda, Google Cloud and Nebius run VMs with the NVIDIA driver and CUDA installed, and ignore `--image`. See [Clouds](https://rrun.041.io/docs/clouds.md). ## Statuses | Status | Meaning | | -------------- | ------------------------------------------------- | | `queued` | Waiting for a machine, or another after a restart | | `provisioning` | A machine is being launched for it | | `starting` | The machine is downloading the bundle | | `running` | The command runs | | `succeeded` | The command exited 0 | | `failed` | See the reason below | | `cancelled` | Someone cancelled it, before or while it ran | A failed job carries a reason, shown after its status as in `failed:no_capacity`, and the command's exit code when there is one: | Reason | Why | | -------------- | --------------------------------------------------------------------- | | `exit` | The command exited non-zero; the exit code is in `EXIT` | | `no_capacity` | No machine was found within `--queue` (default 30 minutes) | | `machine_lost` | The machine stopped answering with no restarts left, or was deleted | | `preempted` | The cloud took the spot machine back, with no restarts left | | `timeout` | The command ran longer than `--timeout` | | `error` | Anything else, such as a command that could not start; see the events | ## Limits on a job | Flag | Default | Effect | | ----------------- | ------- | ---------------------------------------------------------- | | `--timeout ` | none | Kill the command after this long; the job fails `timeout` | | `--queue ` | `30m` | Give up when no machine is found within this long | | `--keep ` | `5m` | Keep the machine idle this long afterwards for another job | | `--restarts ` | `10` | Run again on another machine when the machine fails | Durations are `30s`, `10m`, `6h`, `1d`, `1h30m` or plain seconds. ## Restarts When the machine fails under a started command (the cloud takes a spot machine back, or the machine stops answering), rrun runs the command again on another machine, up to `--restarts` times (default 10). Nothing else restarts it: not the command's own exit (an out-of-memory kill included), not `--timeout`, not a cancel. `--restarts 0` turns restarts off. A restart is the same job, with the same id. It goes back to `queued` with a fresh `--queue` window and skips the offer that failed for 10 minutes; `--timeout` counts again from the new start. Its logs go on after a line such as ```text [rrun] machine m-… was preempted; restarting on another machine (attempt 2 of 11) ``` and `status` shows the attempt. Make the script resumable. Every attempt starts in a fresh directory holding the bundle, with `RRUN_ATTEMPT` set to 1 for the first run, 2 for the first restart, and so on. Save checkpoints to durable storage, such as [Artifacts by 041](https://artifacts.041.io) or a bucket, and load the latest one when there is one. A command that must not run twice takes `--restarts 0`. ## Following and waiting ```bash o41-rrun run ... -- ./train.sh # print the job id and return o41-rrun run ... -f -- ./train.sh # stream logs, exit with the job's code o41-rrun run ... --wait -- ./eval.sh # no logs, print the result, same exit code o41-rrun wait j-4k2m9x7qpz # wait for an existing job o41-rrun cancel j-4k2m9x7qpz # SIGTERM, then SIGKILL after 30 s ``` Ctrl-C while following or waiting stops only the CLI, which exits 130 and prints how to follow again; the job keeps running. A dropped connection is retried with backoff, so a long follow survives a network blip. Exit codes are in [API keys and agents](https://rrun.041.io/docs/api-keys-and-agents.md#exit-codes). ## Logs ```bash o41-rrun logs j-4k2m9x7qpz # everything so far o41-rrun logs j-4k2m9x7qpz --tail 100 # the last 100 lines, then stop o41-rrun logs j-4k2m9x7qpz --tail 100 -f # the last 100, then follow ``` The agent numbers every line of stdout and stderr. They print as they are; rrun's own lines start with `[rrun]`. With `--json`, each line is one JSON object: ```json { "seq": 41, "at": 1792312800123, "stream": "out", "text": "step 100 loss 3.41" } ``` `stream` is `out`, `err` or `sys` (rrun's own). Logs stay readable after the job and the machine are gone: recent lines are served from the job, older ones from the object store, which is rrun's own bucket or [yours](https://rrun.041.io/docs/storage.md). rrun's bucket keeps them 30 days; past that a read prints what is left and a note on stderr. [Your own bucket](https://rrun.041.io/docs/storage.md#retention) keeps them as long as you like. ## Status and telemetry ```bash o41-rrun status j-4k2m9x7qpz ``` Shows the command, who ran it and when, what was requested, the machine it got (cloud, region, GPU, CPUs, RAM, spot) and its price, env and secret names, the limits, the attempt, the events (launches, fall-throughs, failures, restarts) and the latest telemetry: CPU and RAM, and per GPU its utilization, memory, temperature and power, sampled every 15 seconds. `o41-rrun show` is the same command. ```bash o41-rrun telemetry j-4k2m9x7qpz # the whole job o41-rrun telemetry j-4k2m9x7qpz --since 6h # the last six hours o41-rrun telemetry j-4k2m9x7qpz --json # one sample per line ``` `telemetry` prints the samples over time, one row each: time, CPU, RAM, and per GPU its utilization, memory used and total, temperature and power. A range with more than `--points` samples (default 500, at most 2000) is cut into equal time buckets and each one averaged, so a multi-day job reads in one screen; narrow it with `--since` to see every sample. The job's page charts the same numbers with a range picker. Telemetry is kept like logs: 30 days in rrun's bucket, as long as you like in [yours](https://rrun.041.io/docs/storage.md). ## Listing ```bash o41-rrun ls # the newest 20 o41-rrun ls --active # only jobs that have not ended o41-rrun ls --all # every job o41-rrun ls --limit 100 ``` Columns: `ID NAME STATUS AGE GPU CLOUD $/H EXIT`. A job's name defaults to its folder's name; set it with `--name`. --- rrun by 041 documentation. Every page: https://rrun.041.io/llms.txt