# Machines and costs A machine is one cloud instance: a container on Vast and RunPod, a VM on Verda, Google Cloud and Nebius. It runs one job at a time, in your cloud account, billed by the cloud. rrun picks the cheapest one that fits and deletes it when nothing needs it. A [reserved machine](#reserved-machines) is a server of your own that rrun runs jobs on at no cost and never deletes. ## Asking for a machine `run` and `offers` take the same flags. One of `--gpu` and `--cpu-only` is required. | Flag | Meaning | | ------------------- | ------------------------------------------------------------------ | | `--gpu ` | Any one of these, comma separated: `MODEL` or `MODEL:N` for N GPUs | | `--cpu-only` | No GPU | | `--ram ` | Minimum RAM | | `--cpus ` | Minimum vCPUs | | `--vram ` | Minimum memory per GPU | | `--disk ` | Minimum disk (default 50) | | `--spot` | Consider spot offers too; the cheapest offer still wins | | `--cloud ` | Only these clouds or accounts, as `vast` or `vast/lab-b` | | `--max-price <$/h>` | Highest price per hour, in US dollars | | `--cuda ` | Lowest CUDA version the driver must support, such as `12.8` | Each choice is a GPU model and how many of them one machine needs, so a request can range from a single consumer card to a full node: ```bash o41-rrun run --gpu RTX5090 -- python explore.py # one RTX 5090 o41-rrun run --gpu RTX5080:4,A100,H200 --spot -- ./sweep.sh # 4 RTX 5080s, or one A100 or H200 o41-rrun run --gpu B300:8,B200:8 --spot -- torchrun train.py # a whole 8-GPU node ``` rrun prices every choice on every connected cloud account and takes the cheapest. A machine with exactly that many GPUs is launched; an idle machine with more of the same model can take the job too. GPU names are normalized, so `--gpu 5090`, `RTX5090` and `"RTX 5090"` match the same offers. Canonical names look like `RTX5090`, `RTX5080`, `A6000`, `L4`, `T4`, `A100`, `A100-80GB`, `H100`, `H200`, `B200` and `B300`. ## Offers ```bash o41-rrun offers --gpu RTX5090,RTX5080,A6000 --ram 64 --spot ``` Columns: `CLOUD GPU VRAM CPUS RAM $/H SPOT REGION`, where `CLOUD` is the cloud account. An offer is a price for a machine shape on one cloud account right now. `offers` asks every connected account that takes this kind of job at once ([switches](https://rrun.041.io/docs/clouds.md#turning-accounts-on-and-off)) and prints the cheapest 20 (`--limit n` for more or fewer); an account that fails to answer is a warning on stderr, not an error. Reserved machines are not offers: they cost nothing and are taken first when free. Prices are per hour in US dollars; on Verda, Google Cloud and Nebius they include the requested disk. Where the prices come from is in [Clouds](https://rrun.041.io/docs/clouds.md). ## How rrun picks 1. A free [reserved machine](#reserved-machines) that fits, since it costs nothing; then an idle cloud machine that fits. Fitting means one of the GPU models, at least the GPU count, VRAM, RAM, CPUs and disk, the CUDA version, an allowed cloud account, under `--max-price`, running the same image, and spot only when `--spot` is set. The cheapest one wins. 2. Otherwise the cheapest offer across the connected accounts that take this kind of job. When an account has no capacity for it, the next offer is tried; an offer that failed is skipped for 10 minutes. An account that rejects the credentials is skipped for the round. Where a job may run is its placement: | `run` flag | Placement | Takes | | ----------------- | ---------- | ------------------------------------------------------------------------- | | (none) | `any` | A free reserved machine first, otherwise a cloud as above | | `--reserved-only` | `reserved` | Only reserved machines; waits for one up to `--queue`, never launches one | | `--cloud-only` | `cloud` | Only cloud machines, even while a reserved machine is free | A job with `--cloud` or `--image` never takes a reserved machine. A restarted job keeps its placement. One team can keep to cheap spot machines with `--cloud-only --spot` while another runs on the reserved servers with `--reserved-only`. Without `--spot`, only on-demand offers count. With it, spot and on-demand offers compete on price. ## Spot Spot machines cost a fraction of on-demand because the cloud can take them back. When it does, rrun restarts the job on another machine, up to `--restarts` times (10 by default), and fails it with `preempted` only when none are left. The command starts over in a fresh directory, so have it save checkpoints to durable storage and resume from the latest ([Artifacts by 041](https://artifacts.041.io) keeps checkpoints across clouds). See [Restarts](https://rrun.041.io/docs/jobs-and-logs.md#restarts). Vast bids for spot hosts, RunPod has no spot, Verda and Google Cloud have spot VMs, and Nebius has preemptible VMs at a live spot price. See [Clouds](https://rrun.041.io/docs/clouds.md). ## Idle machines and `--keep` When a job ends, its machine idles for that job's `--keep` (default `5m`) so the next job that fits starts in 2–3 s instead of booting a new machine. Then it is deleted. `--keep 0` deletes it as soon as the job ends. An idle machine is billed by the cloud like a busy one. Keep it longer when you are iterating; set `--keep 0` for a one-off. ```bash o41-rrun machines # busy, idle, booting and reserved machines, with IDLE UNTIL o41-rrun machines --all # terminated ones too o41-rrun machines rm m-… # delete one now; its job, if any, fails machine_lost ``` Machine statuses: `provisioning`, `booting`, `idle`, `busy`, `offline` (a reserved machine whose agent is not connected), `terminating`, `terminated`. Deleting a machine needs a signed-in person, not an API key. ## Reserved machines A reserved machine is a server of your own: a Linux x64 or arm64 machine with systemd, in a lab, a rack or any cloud. rrun runs jobs on it one at a time, as it does on cloud VMs, at a price of 0, and never launches, stops or deletes it. ```bash o41-rrun machines register --name lab-1 ``` This prints one command to run on the server, as a user with sudo: ```bash curl -fsSL https://rrun.041.io/install-server.sh | sudo O41_RRUN_MACHINE_KEY=mk_… sh ``` As root, leave out `sudo`. The command holds a key bound to that one machine, shown only now; rrun keeps only its hash. The script checks the key with rrun, installs the current agent under `/var/lib/rrun` after checking its SHA-256, keeps the key in a file only root can read, and runs the agent as the `rrun-agent` systemd service, restarted whenever it stops and started at boot. Run the same command again to update the agent. The key can only connect that machine's agent and download its jobs' folders; it is not an API key, and API keys do not install machines. The web app's Machines page has a Register button that shows the same command. An agent can register a server and set it up over ssh in one go: ```bash ssh lab-1 "$(o41-rrun machines register --name lab-1 --json | jq -r .command)" ``` When its agent connects, the machine reports its hardware: GPU model and count (from `nvidia-smi`), VRAM, CPUs, RAM, free disk and the CUDA version its driver supports. Jobs are matched against that report as they would be against a cloud offer, so `--disk` (default 50 GB) is checked against free disk. Jobs run on the host in a fresh folder; jobs with `--image` never go there. The machine is `idle` or `busy` while its agent is connected and `offline` otherwise; it reconnects by itself after a network drop or a reboot. Keep and idle times do not apply. ```bash o41-rrun machines off m-… # no new jobs; a running job finishes o41-rrun machines on m-… # takes jobs again o41-rrun machines rm m-… # remove it from rrun ``` A reserved machine that is turned off shows as `off`, or `off (busy)` while its last job finishes. An API key may turn machines on and off. If the machine drops mid-job (a reboot, a crash, or ten minutes without word), the job is restarted like any lost machine's, within its placement, up to `--restarts` times. `machines rm` revokes the key: the agent stops for good and logs how to uninstall it, and the machine stays listed with `--all` as `terminated`. Remove the agent from the server with: ```bash curl -fsSL https://rrun.041.io/install-server.sh | sudo sh -s uninstall ``` Registering, turning on and off, and removing are in the audit log. ## The sweeper Every 10 minutes, for each connected cloud account, rrun lists the instances whose names carry that account's rrun prefix (your organization and the account) and deletes any that no machine owns. A launch records its machine before it calls the provider, so a paid instance never outlives its owner. Instances without the prefix are never touched, so two accounts on the same provider account never clean up each other's machines. Reserved machines are never swept. ## Disks Disks go with their machine: - On Vast and RunPod the disk is part of the container. - Google Cloud boot disks are deleted with the VM, and spot VMs are deleted, not stopped, when preempted. - Verda volumes outlive their instance unless deleted with it, and a plain delete only moves them to a trash that counts against the storage quota for 96 hours. rrun names each OS volume after its instance, deletes volumes permanently with the instance, and marks spot volumes for permanent deletion on eviction. The sweeper also deletes any prefixed volume in the trash, owned by no machine, or left by an ended instance. - Nebius boot disks are created with the VM, named after it, and deleted with it. A preempted VM is stopped and keeps its disk until rrun deletes the VM, which it does as soon as it sees the preemption. rrun waits for the VM to go and deletes any of its disks still there, and the sweeper deletes any prefixed disk owned by no machine or left by an ended VM. Every deletion is in the audit log (`o41-rrun audit`). ## Limits - At most 16 live cloud machines per organization; reserved machines do not count. More jobs wait in `queued` until one frees up. - `--max-price` caps a single machine's hourly price. - `--timeout` caps how long a command runs; `--queue` caps how long a job waits for a machine. ## Audit log ```bash o41-rrun audit --limit 100 ``` Every launch, job end, cancel, machine deletion, reserved machine change, and change to clouds, their switches, storage and API keys, with who did it. It needs a signed-in person. --- rrun by 041 documentation. Every page: https://rrun.041.io/llms.txt