# How it works

You describe the machine you need and the command to run. rrun finds the
machine, runs the command, streams the logs back and cleans up afterwards. There
is nothing for you to host or keep running.

## A job, start to finish

1. **Pack.** `o41-rrun run` packs the folder and uploads it, unless rrun already
   has that exact content. The job is `queued`.
2. **Find a machine.** If one of your
   [reserved machines](https://rrun.041.io/docs/machines-and-costs.md#reserved-machines) is free and
   fits, the job takes it, since it costs nothing; next comes an idle cloud
   machine that fits. Either starts the job in 2–3 seconds. Otherwise rrun asks
   every connected cloud account that takes this kind of job for offers and
   launches the cheapest one that fits (`provisioning`). A cloud without
   capacity falls through to the next offer. `--reserved-only` waits for a
   reserved machine instead, and `--cloud-only` never takes one. When nothing
   fits, rrun keeps trying until `--queue` runs out (30 minutes by default),
   then the job fails with `no_capacity`.
3. **Boot.** The new machine starts a small agent that connects back to rrun,
   downloads your folder into a fresh directory and runs the command
   (`starting`, then `running`). A cold machine takes 30–240 seconds from launch
   to running, depending on the cloud and the image.
4. **Run.** Logs and GPU, CPU and RAM telemetry stream back while the command
   runs. `o41-rrun logs -f` and the job's page in the web app show them live.
5. **Finish.** The command's exit code ends the job as `succeeded`, `failed` or
   `cancelled`.
6. **Keep warm, then delete.** A cloud machine waits for the job's `--keep` (5
   minutes by default) in case your next job fits, then it is deleted together
   with its disks. A reserved machine simply waits for the next job.

If a machine never comes up, the job goes back to step 2 and tries another
offer. If the machine fails while the command runs, or is reported bad, the job
also goes back to step 2, keeping its placement, and the command runs again, up
to `--restarts` times (10 by default; see
[Restarts](https://rrun.041.io/docs/jobs-and-logs.md#restarts)). A cancel at any point ends the job for
good.

## When things go wrong

- **Lost machine.** A cloud machine that stops reporting is deleted, and its job
  restarts on another machine. A reserved machine that stops reporting, or
  reboots, mid-job loses that job the same way, and is `offline` until its agent
  is back. With no restarts left, the job fails with `machine_lost`, or
  `preempted` when the cloud took a spot machine back.
- **Bad machine.** A machine can look healthy to rrun and still be broken: a
  hung driver install, a GPU that does not work. A command that exits `75`, or
  `o41-rrun cancel --bad-host`, says so; rrun deletes the machine, skips its
  host, and restarts the job elsewhere as for a lost machine
  ([A bad machine](https://rrun.041.io/docs/jobs-and-logs.md#a-bad-machine)).
- **Timeout.** `--timeout` stops the command and fails the job with `timeout`.
- **Cancel.** The command gets SIGTERM, then SIGKILL after 30 seconds.
- **Leftovers.** rrun regularly checks each cloud account for machines and disks
  named for your organization and that account that no job owns, and deletes
  them. See [Machines and costs](https://rrun.041.io/docs/machines-and-costs.md#the-sweeper).

## Logs

Logs and telemetry are kept in rrun's storage for 30 days, or in
[your own bucket](https://rrun.041.io/docs/storage.md) as long as you like, and stay readable after the
machine is gone.

## Security

- Cloud credentials are encrypted and only used to talk to that cloud. Google
  Cloud can connect without a key at all; see [Clouds](https://rrun.041.io/docs/clouds.md#google-cloud).
- Each machine gets its own token, good only for its own job, and the token ends
  with the machine. A reserved machine's key is bound to that machine and works
  until the machine is removed; rrun keeps only its hash.
- An organization API key can manage jobs only. See
  [API keys and agents](https://rrun.041.io/docs/api-keys-and-agents.md).

---

rrun by 041 documentation. Every page: https://rrun.041.io/llms.txt