Machines and costs

A machine is one cloud instance: a container on Vast and RunPod, a VM on Verda, Google Cloud and Nebius. It runs one job at a time, in your cloud account, billed by the cloud. rrun picks the cheapest one that fits and deletes it when nothing needs it. A reserved machine is a server of your own that rrun runs jobs on at no cost and never deletes.

Asking for a machine

run and offers take the same flags. One of --gpu and --cpu-only is required.

Flag	Meaning
`--gpu <choices>`	Any one of these, comma separated: `MODEL` or `MODEL:N` for N GPUs
`--cpu-only`	No GPU
`--ram <GB>`	Minimum RAM
`--cpus <n>`	Minimum vCPUs
`--vram <GB>`	Minimum memory per GPU
`--disk <GB>`	Minimum disk (default 50)
`--spot`	Consider spot offers too; the cheapest offer still wins
`--cloud <list>`	Only these clouds or accounts, as `vast` or `vast/lab-b`
`--max-price <$/h>`	Highest price per hour, in US dollars
`--cuda <version>`	Lowest CUDA version the driver must support, such as `12.8`

Each choice is a GPU model and how many of them one machine needs, so a request can range from a single consumer card to a full node:

o41-rrun run --gpu RTX5090 -- python explore.py            # one RTX 5090
o41-rrun run --gpu RTX5080:4,A100,H200 --spot -- ./sweep.sh # 4 RTX 5080s, or one A100 or H200
o41-rrun run --gpu B300:8,B200:8 --spot -- torchrun train.py # a whole 8-GPU node

rrun prices every choice on every connected cloud account and takes the cheapest. A machine with exactly that many GPUs is launched; an idle machine with more of the same model can take the job too.

GPU names are normalized, so --gpu 5090, RTX5090 and "RTX 5090" match the same offers. Canonical names look like RTX5090, RTX5080, A6000, L4, T4, A100, A100-80GB, H100, H200, B200 and B300.

Offers

o41-rrun offers --gpu RTX5090,RTX5080,A6000 --ram 64 --spot

Columns: CLOUD GPU VRAM CPUS RAM $/H SPOT REGION, where CLOUD is the cloud account. An offer is a price for a machine shape on one cloud account right now. offers asks every connected account that takes this kind of job at once (switches) and prints the cheapest 20 (--limit n for more or fewer); an account that fails to answer is a warning on stderr, not an error. Reserved machines are not offers: they cost nothing and are taken first when free. Prices are per hour in US dollars; on Verda, Google Cloud and Nebius they include the requested disk. Where the prices come from is in Clouds.

How rrun picks

A free reserved machine that fits, since it costs nothing; then an idle cloud machine that fits. Fitting means one of the GPU models, at least the GPU count, VRAM, RAM, CPUs and disk, the CUDA version, an allowed cloud account, under --max-price, running the same image, and spot only when --spot is set. The cheapest one wins.
Otherwise the cheapest offer across the connected accounts that take this kind of job. When an account has no capacity for it, the next offer is tried; an offer that failed is skipped for 10 minutes. An account that rejects the credentials is skipped for the round.

Where a job may run is its placement:

`run` flag	Placement	Takes
(none)	`any`	A free reserved machine first, otherwise a cloud as above
`--reserved-only`	`reserved`	Only reserved machines; waits for one up to `--queue`, never launches one
`--cloud-only`	`cloud`	Only cloud machines, even while a reserved machine is free

A job with --cloud or --image never takes a reserved machine. A restarted job keeps its placement. One team can keep to cheap spot machines with --cloud-only --spot while another runs on the reserved servers with --reserved-only.

Without --spot, only on-demand offers count. With it, spot and on-demand offers compete on price.

Spot

Spot machines cost a fraction of on-demand because the cloud can take them back. When it does, rrun restarts the job on another machine, up to --restarts times (10 by default), and fails it with preempted only when none are left. The command starts over in a fresh directory, so have it save checkpoints to durable storage and resume from the latest (Artifacts by 041 keeps checkpoints across clouds). See Restarts.

Vast bids for spot hosts, RunPod has no spot, Verda and Google Cloud have spot VMs, and Nebius has preemptible VMs at a live spot price. See Clouds.

Idle machines and `--keep`

When a job ends, its machine idles for that job's --keep (default 5m) so the next job that fits starts in 2–3 s instead of booting a new machine. Then it is deleted. --keep 0 deletes it as soon as the job ends.

An idle machine is billed by the cloud like a busy one. Keep it longer when you are iterating; set --keep 0 for a one-off.

o41-rrun machines          # busy, idle, booting and reserved machines, with IDLE UNTIL
o41-rrun machines --all    # terminated ones too
o41-rrun machines rm m-…   # delete one now; its job, if any, fails machine_lost

Machine statuses: provisioning, booting, idle, busy, offline (a reserved machine whose agent is not connected), terminating, terminated. Deleting a machine needs a signed-in person, not an API key.

Reserved machines

A reserved machine is a server of your own: a Linux x64 or arm64 machine with systemd, in a lab, a rack or any cloud. rrun runs jobs on it one at a time, as it does on cloud VMs, at a price of 0, and never launches, stops or deletes it.

o41-rrun machines register --name lab-1

This prints one command to run on the server, as a user with sudo:

curl -fsSL https://rrun.041.io/install-server.sh | sudo O41_RRUN_MACHINE_KEY=mk_… sh

As root, leave out sudo. The command holds a key bound to that one machine, shown only now; rrun keeps only its hash. The script checks the key with rrun, installs the current agent under /var/lib/rrun after checking its SHA-256, keeps the key in a file only root can read, and runs the agent as the rrun-agent systemd service, restarted whenever it stops and started at boot. Run the same command again to update the agent. The key can only connect that machine's agent and download its jobs' folders; it is not an API key, and API keys do not install machines. The web app's Machines page has a Register button that shows the same command.

An agent can register a server and set it up over ssh in one go:

ssh lab-1 "$(o41-rrun machines register --name lab-1 --json | jq -r .command)"

When its agent connects, the machine reports its hardware: GPU model and count (from nvidia-smi), VRAM, CPUs, RAM, free disk and the CUDA version its driver supports. Jobs are matched against that report as they would be against a cloud offer, so --disk (default 50 GB) is checked against free disk. Jobs run on the host in a fresh folder; jobs with --image never go there. The machine is idle or busy while its agent is connected and offline otherwise; it reconnects by itself after a network drop or a reboot. Keep and idle times do not apply.

o41-rrun machines off m-…   # no new jobs; a running job finishes
o41-rrun machines on m-…    # takes jobs again
o41-rrun machines rm m-…    # remove it from rrun

A reserved machine that is turned off shows as off, or off (busy) while its last job finishes. An API key may turn machines on and off. If the machine drops mid-job (a reboot, a crash, or ten minutes without word), the job is restarted like any lost machine's, within its placement, up to --restarts times.

machines rm revokes the key: the agent stops for good and logs how to uninstall it, and the machine stays listed with --all as terminated. Remove the agent from the server with:

curl -fsSL https://rrun.041.io/install-server.sh | sudo sh -s uninstall

Registering, turning on and off, and removing are in the audit log.

The sweeper

Every 10 minutes, for each connected cloud account, rrun lists the instances whose names carry that account's rrun prefix (your organization and the account) and deletes any that no machine owns. A launch records its machine before it calls the provider, so a paid instance never outlives its owner. Instances without the prefix are never touched, so two accounts on the same provider account never clean up each other's machines. Reserved machines are never swept.

Disks

Disks go with their machine:

On Vast and RunPod the disk is part of the container.
Google Cloud boot disks are deleted with the VM, and spot VMs are deleted, not stopped, when preempted.
Verda volumes outlive their instance unless deleted with it, and a plain delete only moves them to a trash that counts against the storage quota for 96 hours. rrun names each OS volume after its instance, deletes volumes permanently with the instance, and marks spot volumes for permanent deletion on eviction. The sweeper also deletes any prefixed volume in the trash, owned by no machine, or left by an ended instance.
Nebius boot disks are created with the VM, named after it, and deleted with it. A preempted VM is stopped and keeps its disk until rrun deletes the VM, which it does as soon as it sees the preemption. rrun waits for the VM to go and deletes any of its disks still there, and the sweeper deletes any prefixed disk owned by no machine or left by an ended VM.

Every deletion is in the audit log (o41-rrun audit).

Limits

At most 16 live cloud machines per organization; reserved machines do not count. More jobs wait in queued until one frees up.
--max-price caps a single machine's hourly price.
--timeout caps how long a command runs; --queue caps how long a job waits for a machine.

Audit log

o41-rrun audit --limit 100

Every launch, job end, cancel, machine deletion, reserved machine change, and change to clouds, their switches, storage and API keys, with who did it. It needs a signed-in person.