Jobs and logs

A job is a folder (the bundle), a command, what machine it needs, and its env and secrets. When its machine fails under it, the job restarts on another machine (Restarts); otherwise it runs once, and a retry is a new job. Job ids look like j-4k2m9x7qpz.

What gets packed

o41-rrun run packs the current folder, or the one -C <path> names:

In a git work tree: the tracked files, plus untracked files .gitignore does not exclude. What git would commit, uncommitted edits included.
Anywhere else: everything except .git.

Symlinks stay symlinks and the executable bit is kept; submodules and sockets are skipped with a note. A bundle is at most 95 MiB gzipped, and a folder over 2 GiB is refused before packing. Leave data, checkpoints and environments out with .gitignore and fetch them inside the job.

Bundles are stored by their sha256, so running the same folder again uploads nothing.

The command

Everything after the flags is the command. Flags end at the first argument that is not a flag, or at --:

o41-rrun run --cpu-only python train.py --lr 3     # --lr 3 goes to train.py
o41-rrun run --gpu L4 -- 'python prep.py && ls out' # one argument: a shell script

One argument runs as a shell script as given. Several are quoted into one command line. Either way it runs under sh -c in a fresh directory holding the unpacked bundle, in its own process group. The command is at most 16 KiB.

Env and secrets

o41-rrun run --gpu A6000 --env DATASET=small --env WANDB_MODE --secret HF_TOKEN -- ./train.sh

--env K=V sets a variable. --env K copies K from your shell and fails when it is not set. Repeatable.
--secret works the same way, but the value is sealed at rest, never shown back (status lists only names) and deleted when the job ends.
Names are letters, digits and _, not starting with a digit. Env and secrets together are at most 64 KiB.

The command also gets RRUN_JOB_ID and RRUN_ATTEMPT (Restarts). The machine's own rrun token is never in the command's environment.

Images

On the container clouds (Vast and RunPod), --image <ref> picks the container image. Without it, rrun uses the cloud's Ubuntu image with CUDA that the host's driver supports. Verda, Google Cloud and Nebius run VMs with the NVIDIA driver and CUDA installed, and ignore --image. See Clouds.

Statuses

Status	Meaning
`queued`	Waiting for a machine, or another after a restart
`provisioning`	A machine is being launched for it
`starting`	The machine is downloading the bundle
`running`	The command runs
`succeeded`	The command exited 0
`failed`	See the reason below
`cancelled`	Someone cancelled it, before or while it ran

A failed job carries a reason, shown after its status as in failed:no_capacity, and the command's exit code when there is one:

Reason	Why
`exit`	The command exited non-zero; the exit code is in `EXIT`
`no_capacity`	No machine was found within `--queue` (default 30 minutes)
`machine_lost`	The machine stopped answering with no restarts left, or was deleted
`preempted`	The cloud took the spot machine back, with no restarts left
`timeout`	The command ran longer than `--timeout`
`error`	Anything else, such as a command that could not start; see the events

Limits on a job

Flag	Default	Effect
`--timeout <dur>`	none	Kill the command after this long; the job fails `timeout`
`--queue <dur>`	`30m`	Give up when no machine is found within this long
`--keep <dur>`	`5m`	Keep the machine idle this long afterwards for another job
`--restarts <n>`	`10`	Run again on another machine when the machine fails

Durations are 30s, 10m, 6h, 1d, 1h30m or plain seconds.

Restarts

When the machine fails under a started command (the cloud takes a spot machine back, or the machine stops answering), rrun runs the command again on another machine, up to --restarts times (default 10). Nothing else restarts it: not the command's own exit (an out-of-memory kill included), not --timeout, not a cancel. --restarts 0 turns restarts off.

A restart is the same job, with the same id. It goes back to queued with a fresh --queue window and skips the offer that failed for 10 minutes; --timeout counts again from the new start. Its logs go on after a line such as

[rrun] machine m-… was preempted; restarting on another machine (attempt 2 of 11)

and status shows the attempt.

Make the script resumable. Every attempt starts in a fresh directory holding the bundle, with RRUN_ATTEMPT set to 1 for the first run, 2 for the first restart, and so on. Save checkpoints to durable storage, such as Artifacts by 041 or a bucket, and load the latest one when there is one. A command that must not run twice takes --restarts 0.

Following and waiting

o41-rrun run ... -- ./train.sh      # print the job id and return
o41-rrun run ... -f -- ./train.sh   # stream logs, exit with the job's code
o41-rrun run ... --wait -- ./eval.sh # no logs, print the result, same exit code
o41-rrun wait j-4k2m9x7qpz           # wait for an existing job
o41-rrun cancel j-4k2m9x7qpz         # SIGTERM, then SIGKILL after 30 s

Ctrl-C while following or waiting stops only the CLI, which exits 130 and prints how to follow again; the job keeps running. A dropped connection is retried with backoff, so a long follow survives a network blip. Exit codes are in API keys and agents.

Logs

o41-rrun logs j-4k2m9x7qpz              # everything so far
o41-rrun logs j-4k2m9x7qpz --tail 100   # the last 100 lines, then stop
o41-rrun logs j-4k2m9x7qpz --tail 100 -f # the last 100, then follow

The agent numbers every line of stdout and stderr. They print as they are; rrun's own lines start with [rrun]. With --json, each line is one JSON object:

{
  "seq": 41,
  "at": 1792312800123,
  "stream": "out",
  "text": "step 100 loss 3.41"
}

stream is out, err or sys (rrun's own). Logs stay readable after the job and the machine are gone: recent lines are served from the job, older ones from the object store, which is rrun's own bucket or yours. rrun's bucket keeps them 30 days; past that a read prints what is left and a note on stderr. Your own bucket keeps them as long as you like.

Status and telemetry

o41-rrun status j-4k2m9x7qpz

Shows the command, who ran it and when, what was requested, the machine it got (cloud, region, GPU, CPUs, RAM, spot) and its price, env and secret names, the limits, the attempt, the events (launches, fall-throughs, failures, restarts) and the latest telemetry: CPU and RAM, and per GPU its utilization, memory, temperature and power, sampled every 15 seconds. o41-rrun show is the same command.

o41-rrun telemetry j-4k2m9x7qpz               # the whole job
o41-rrun telemetry j-4k2m9x7qpz --since 6h    # the last six hours
o41-rrun telemetry j-4k2m9x7qpz --json        # one sample per line

telemetry prints the samples over time, one row each: time, CPU, RAM, and per GPU its utilization, memory used and total, temperature and power. A range with more than --points samples (default 500, at most 2000) is cut into equal time buckets and each one averaged, so a multi-day job reads in one screen; narrow it with --since to see every sample. The job's page charts the same numbers with a range picker. Telemetry is kept like logs: 30 days in rrun's bucket, as long as you like in yours.

Listing

o41-rrun ls               # the newest 20
o41-rrun ls --active      # only jobs that have not ended
o41-rrun ls --all         # every job
o41-rrun ls --limit 100

Columns: ID NAME STATUS AGE GPU CLOUD $/H EXIT. A job's name defaults to its folder's name; set it with --name.