# Command line `o41-rrun` runs a folder's command on the cheapest machine that fits and streams its logs back. Install it with: ```bash curl -fsSL https://rrun.041.io/install.sh | sh ``` It checks the download against the release's SHA-256 before installing; [`o41-rrun update`](#update) moves to the newest release later. `o41-rrun help`, or `o41-rrun --help`, prints the same reference. ## Commands | Command | Does | | ------------------------------------------------------- | ---------------------------------------------------- | | [`run`](#run) | Pack a folder, run a command on the cheapest machine | | [`ls`](#ls) | List jobs, newest first | | [`status `](#status), `show ` | A job's spec, machine, events and latest telemetry | | [`logs `](#logs) | Print a job's logs; `-f` follows until it ends | | [`telemetry `](#telemetry) | A job's machine telemetry over time | | [`wait `](#wait) | Wait for a job to end; exit with its exit code | | [`cancel `](#cancel) | Cancel a job | | [`offers`](#offers) | What a machine would cost on every connected cloud | | [`machines`](#machines), `register`, `on`, `off`, `rm` | Cloud and reserved machines; add your own servers | | [`clouds`](#clouds), `set`, `on`, `off`, `rm` | Cloud accounts and the jobs each takes | | [`storage`](#storage), `storage set`, `storage rm` | Where job logs are stored, and for how long | | [`api-keys`](#api-keys), `api-keys create`, `revoke` | Organization API keys | | [`audit`](#audit) | The organization's audit log | | [`login`](#login), `logout`, `whoami`, `org`, `org use` | Sign-in and organization | | [`feedback`](#feedback) | Send feedback or a bug report to the 041 team | | [`update`](#update) | Replace this binary with the newest release | ## Global flags | Flag | Effect | | ----------------- | --------------------------------------------------------------------- | | `--json` | One JSON document on stdout; log streams are JSON Lines | | `--url ` | The rrun server (default: `O41_RRUN_URL`, else `https://rrun.041.io`) | | `-h`, `--help` | Help for a command, as in `o41-rrun run --help` | | `-V`, `--version` | Print the version (before the command) | Global flags go before the command or among its flags. Output and exit codes are in [API keys and agents](https://rrun.041.io/docs/api-keys-and-agents.md#output). ## run ```text o41-rrun run (--gpu | --cpu-only) [options] [--] ``` Packs the folder, uploads it unless the server has it, queues the job and prints its id. One argument runs as a shell script (`'python a.py && ls'`); several are quoted into one command line. Flags end at the first argument, so `o41-rrun run --cpu-only python train.py --lr 3` passes `--lr 3` to the program. See [Jobs and logs](https://rrun.041.io/docs/jobs-and-logs.md). Job: | Flag | Default | Meaning | | -------------------- | --------------- | --------------------------------------------------------------------------------------------- | | `-C`, `--dir ` | `.` | Folder to pack and run in | | `--name ` | the folder name | Job name, at most 128 characters | | `--env K=V` | | Environment variable; `--env K` copies K from here. Repeatable. | | `--secret K=V` | | Like `--env`, but sealed and never shown back. Repeatable. | | `--image ` | the cloud's | Container image on container clouds (vast, runpod) | | `--timeout ` | `0` (never) | Kill the command after this long | | `--keep ` | `5m` | Keep the machine idle this long for the next job | | `--queue ` | `30m` | Give up when no machine is found within this long | | `--restarts ` | `10` | When the machine fails, run again on another; `0` never | | `--reserved-only` | | Only your [reserved machines](https://rrun.041.io/docs/machines-and-costs.md#reserved-machines); wait within `--queue` | | `--cloud-only` | | Never a reserved machine | | `-f`, `--follow` | | Stream logs; exit with the job's exit code | | `--wait` | | Wait quietly; print the result; exit with the job's code | Machine (also for [`offers`](#offers)): | Flag | Default | Meaning | | ------------------- | ------------- | ------------------------------------------------------------------------------------------------- | | `--gpu ` | | Any one of these, comma separated: `MODEL` for one GPU, `MODEL:N` for N, as in `RTX5080:4,H100:8` | | `--cpu-only` | | No GPU. One of `--gpu` and `--cpu-only` is required. | | `--ram ` | | Minimum RAM | | `--cpus ` | | Minimum vCPUs | | `--vram ` | | Minimum memory per GPU | | `--disk ` | `50` | Minimum disk | | `--spot` | | Consider spot offers too; the cheapest wins | | `--cloud ` | all connected | Only these clouds (`vast`: all its accounts) or accounts (`vast/lab-b`); no reserved machines | | `--max-price <$/h>` | no limit | Highest price per hour | | `--cuda ` | any | Lowest CUDA version the driver must support, such as `12.8` | Durations: `30s`, `10m`, `6h`, `1d`, `1h30m` or seconds. `--vram` needs `--gpu`. Without `--reserved-only` or `--cloud-only`, a free reserved machine that fits takes the job first, since it costs nothing; otherwise the cheapest cloud offer does. `--reserved-only` excludes `--cloud-only`, `--cloud` and `--image`. A restarted command sees `RRUN_ATTEMPT` (1, 2, ...) and should resume from a checkpoint; see [Restarts](https://rrun.041.io/docs/jobs-and-logs.md#restarts). ```bash o41-rrun run --gpu RTX5090,RTX5080,A6000 --ram 64 --spot -f -- ./train.sh o41-rrun run --cpu-only -C ./etl --env DATASET=small -- 'python prep.py && ls out' o41-rrun run --gpu RTX5080:4,A100,H200 --spot -- python sweep.py o41-rrun run --gpu B300:8,B200:8 --spot --secret HF_TOKEN --timeout 6h -- torchrun train.py ``` By default it returns at once. In a terminal it prints one line, ` queued. Follow: o41-rrun logs -f`; when stdout is captured, as in `id=$(o41-rrun run ...)`, stdout is just the id and the hint goes to stderr. With `--json` it prints the job as JSON. `-f` follows the logs and `--wait` waits. ## ls ```text o41-rrun ls [--active | --all] [--limit ] ``` | Flag | Meaning | | ------------- | ---------------------------------- | | `--active` | Only jobs that have not ended | | `--all` | Every job (default: the newest 20) | | `--limit ` | At most n jobs | Columns: `ID NAME STATUS AGE GPU CLOUD $/H EXIT`. `CLOUD` is the cloud account, such as `vast/default`, or `reserved`. A failure's reason follows its status (`failed:no_capacity`) unless the command exited non-zero. ## status ```text o41-rrun status (alias: o41-rrun show ) ``` The job's id, name, status, exit code, command, who created it and when, start and end, what it requested, the machine and its price, where it may run (placement), image, env and secret names, limits, bundle hash and log line count; then its events and the latest telemetry. ## logs ```text o41-rrun logs [-f] [--tail ] ``` | Flag | Meaning | | ---------------- | --------------------------------------------------------- | | `-f`, `--follow` | Keep printing until the job ends; exit with its exit code | | `--tail ` | Start with the last n lines | stdout and stderr lines print as they are; rrun's own lines start with `[rrun]`. With `--json`, one line per log line: `{"seq","at","stream","text"}`. Lines that rrun's bucket deleted after 30 days are skipped with one note on stderr. ## telemetry ```text o41-rrun telemetry [--since ] [--points ] ``` | Flag | Meaning | | --------------- | ---------------------------------------------------------------------- | | `--since ` | Only the last ``, such as `1h` or `6h` (default: the whole job) | | `--points ` | At most n rows (default 500, at most 2000); longer ranges are averaged | One row per sample: `TIME CPU RAM`, then per GPU its utilization, memory used and total, temperature and power (when the GPU reports them). The agent samples every 15 seconds. With `--json`, one sample per line: `{"at","cpuPercent","ramUsedGb","ramTotalGb","gpus":[{"utilPercent","memUsedGb","memTotalGb","tempC","powerW"}]}`. Samples rrun's bucket deleted after 30 days are left out with a note on stderr. ## wait ```text o41-rrun wait ``` Prints the job's result line (with `--json`, the job) and exits 0 when it succeeded, the command's exit code when it exited non-zero, 1 for other failures and 130 when cancelled. ## cancel ```text o41-rrun cancel ``` The command gets SIGTERM, then SIGKILL after 30 seconds. ## offers ```text o41-rrun offers (--gpu | --cpu-only) [machine flags] [--limit ] ``` Takes the [machine flags of `run`](#run), plus `--limit ` (default 20). Columns: `CLOUD GPU VRAM CPUS RAM $/H SPOT REGION`, cheapest first; `CLOUD` is the account. Only accounts that take this kind of job answer ([`clouds on` and `off`](#clouds)). A cloud that fails to answer is a warning on stderr. ```bash o41-rrun offers --gpu RTX5090,A6000 --ram 64 --spot o41-rrun offers --gpu B300:8,B200:8,H200:8 --spot ``` ## machines ```text o41-rrun machines [--all] o41-rrun machines register [--name ] o41-rrun machines on o41-rrun machines off o41-rrun machines rm ``` `--all` includes terminated machines. Columns: `ID STATUS CLOUD GPU CPUS RAM $/H SPOT WHERE JOB AGE IDLE UNTIL`. `CLOUD` is the cloud account, or `reserved`; `WHERE` is the region, or a reserved machine's name. A reserved machine's status is `offline`, `idle` or `busy`, and `off` when it is turned off (`off (busy)` while its last job finishes). - `machines register` adds a server of your own as a [reserved machine](https://rrun.041.io/docs/machines-and-costs.md#reserved-machines) and prints one command to run on it. In a terminal a line before it explains it; when stdout is captured, stdout is just the command. With `--json`: `{"id","key","command","machine"}`. `--name` defaults to the server's hostname. - `machines off` stops new jobs from landing on a reserved machine; a running job finishes. `machines on` lets it take jobs again. - `machines rm` deletes a cloud machine now, or removes a reserved one: its key stops working and its agent stops. Either way its job, if any, fails with `machine_lost`. `register` and `rm` need a signed-in person; `on` and `off` also take an API key. ```bash ssh lab-1 "$(o41-rrun machines register --name lab-1 --json | jq -r .command)" ``` ## clouds ```text o41-rrun clouds o41-rrun clouds set [--name ] [flags] o41-rrun clouds on [--gpu | --cpu] o41-rrun clouds off [--gpu | --cpu] o41-rrun clouds rm / ``` Each connection is a cloud account, `/`. `clouds` lists them with columns `ACCOUNT IDENTITY GPU CPU MACHINES UPDATED`. `clouds set` connects `/default`, or `/` with `--name` (lowercase letters, digits and dashes); set it again to replace the credentials. `clouds rm` takes the account id and needs the account to have no machines. `clouds off --gpu` stops GPU jobs from launching there, `--cpu` stops CPU-only jobs, and neither flag stops both; `clouds on` turns them back on. A cloud name, as in `clouds off vast`, sets every account of that cloud. Running machines are not touched. See [Clouds](https://rrun.041.io/docs/clouds.md#turning-accounts-on-and-off). Without flags, `clouds set` reads credentials from the usual environment variables and files: | Cloud | Flags | Then | | -------- | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `vast` | `--api-key` | `VAST_API_KEY`, `~/.config/vastai/vast_api_key`, `~/.vast_api_key` | | `runpod` | `--api-key` | `RUNPOD_API_KEY`, `~/.runpod/config.toml` | | `verda` | `--client-id`, `--client-secret` | `VERDA_CLIENT_ID` and `VERDA_CLIENT_SECRET`, `~/.verda/config.json` | | `gcp` | `--workload-identity-provider`, `--service-account`, `--project` | Keyless; see [Clouds](https://rrun.041.io/docs/clouds.md#google-cloud) | | `gcp` | `--credentials ` | `GOOGLE_APPLICATION_CREDENTIALS`, `~/.config/gcloud/application_default_credentials.json` | | `nebius` | `--tenant-id`, `--service-account-id` | Keyless only; see [Clouds](https://rrun.041.io/docs/clouds.md#nebius) | For `gcp`, `--zones us-central1-a,europe-west4-a` limits the zones. The server checks the credentials with the provider and seals them; the CLI prints only where they came from. `clouds`, `set` and `rm` need a signed-in person; `on` and `off` also take an API key, so a script can steer where work goes. ## storage ```text o41-rrun storage o41-rrun storage set --endpoint --bucket [--region ] [--prefix

] o41-rrun storage rm ``` `storage set` takes keys from `--access-key-id` and `--secret-access-key`, else `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. `storage rm` goes back to rrun's own bucket. rrun's bucket keeps logs, telemetry and uploaded folders for 30 days; your own bucket keeps logs and telemetry as long as you like, since rrun never deletes from it. See [Log storage](https://rrun.041.io/docs/storage.md). ```bash o41-rrun storage set --endpoint https://t3.storage.dev --bucket logs --region auto --prefix rrun/ ``` ## api-keys ```text o41-rrun api-keys o41-rrun api-keys create o41-rrun api-keys revoke ``` `create` prints the secret, once, alone on stdout: ```bash export O41_RRUN_API_KEY=$(o41-rrun api-keys create ci) ``` An API key can run, list, follow and cancel jobs, see offers, list machines, and turn cloud accounts and reserved machines on and off. Connecting clouds, registering and removing machines, storage, keys and the audit log need `o41-rrun login`. See [API keys and agents](https://rrun.041.io/docs/api-keys-and-agents.md). ## audit ```text o41-rrun audit [--limit ] (default 50) ``` Columns: `AT ACTOR ACTION TARGET DETAIL`. ## login ```text o41-rrun login [--org ] o41-rrun logout o41-rrun whoami o41-rrun org o41-rrun org use ``` - `login` prints a link and a code; open the link on any device (works over SSH). The sign-in is saved per server in `~/.config/o41-rrun/auth.json` and refreshed as needed. `--org` picks the organization to act in; with one organization it is picked for you. - `logout` deletes the saved sign-in and revokes its tokens. - `whoami` shows the server, the credential and where it came from, the user and the organization in use. Its organization id is the one the keyless [Google Cloud](https://rrun.041.io/docs/clouds.md#without-keys-recommended) and [Nebius](https://rrun.041.io/docs/clouds.md#setup) setups need. - `org` lists your organizations; `*` marks the current one. `org use` picks the one the saved sign-in acts in. ## feedback ```bash o41-rrun feedback "please support Lambda Cloud" --reply-to me@mycompany.com o41-rrun feedback --bug "logs -f stops after an hour on j-4k2m9x7qpz" ``` Sends the message to the 041 team with the CLI's version, OS, server URL and sign-in attached, never a key. It needs no sign-in. `--bug` marks a bug report; `--reply-to` takes an email address or a URL, such as an issue, for the answer. The web app's Feedback page does the same. ## update ```bash o41-rrun update # install the newest release o41-rrun update --check # only say whether this binary is the newest ``` Downloads the newest release for this platform, checks it against the release's SHA-256 and renames it over this binary, through any symlink. A download that does not match leaves the binary as it was. The release comes from the server `--url` or `O41_RRUN_URL` names. `update` needs write access to the binary's directory; without it, run the install script again. ```console $ o41-rrun update Updated /home/me/.local/bin/o41-rrun to o41-rrun 0.1.0-4f9a2c1e. ``` ## Environment | Variable | Effect | | ------------------ | --------------------------------------------------------------- | | `O41_RRUN_API_KEY` | Organization API key (`ak_…`); wins over the saved login | | `O41_RRUN_URL` | The rrun server | | `O41_RRUN_TOKEN` | A raw bearer token, with `O41_RRUN_ORG` naming its organization | | `O41_RRUN_ORG` | The organization to act in, overriding the saved login's | | `XDG_CONFIG_HOME` | The sign-in is saved in `$XDG_CONFIG_HOME/o41-rrun/auth.json` | --- rrun by 041 documentation. Every page: https://rrun.041.io/llms.txt