# Log storage

rrun's built-in storage keeps job logs, telemetry and uploaded folders for 30
days, then deletes them. To keep logs and telemetry longer, set your own
S3-compatible bucket with `o41-rrun storage set` or the web app's Storage page;
rrun never cleans it up.

```bash
o41-rrun storage          # where logs go now, and for how long
o41-rrun storage set --endpoint https://t3.storage.dev --bucket logs --region auto --prefix rrun/
o41-rrun storage rm       # back to rrun's bucket
```

A job's log lines are buffered in the job and flushed as gzipped JSON Lines
chunks: at 1 MiB, every 30 seconds, and when the job ends. Telemetry flushes the
same way in chunks of about 15 minutes. The newest log lines and the last two
hours of telemetry also stay with the job, so tails and recent charts are fast.

## Retention

| Where                           | What                       | Kept                                |
| ------------------------------- | -------------------------- | ----------------------------------- |
| rrun's own bucket (the default) | Logs and telemetry         | 30 days after each chunk is written |
| Your own bucket                 | Logs and telemetry         | As long as your bucket keeps them   |
| rrun's own bucket, always       | Uploaded folders (bundles) | 30 days after the upload            |

rrun never deletes from your bucket; set a lifecycle rule there if you want one.

Reading past the retention is not an error: `o41-rrun logs`,
`o41-rrun telemetry` and the job's page show what is left and say that older
data was deleted. A long job's newest log lines and its last two hours of
telemetry stay with the job, so they outlive the chunks.

From a week before its 30 days are up, an uploaded folder counts as not
uploaded, so the next `o41-rrun run` uploads it again and its 30 days start
over. A job that waits or restarts past its folder's 30 days fails with a
message saying the bundle expired; run it again to upload the folder.

## Your own bucket

`storage set` takes:

| Flag                        | Meaning                                              |
| --------------------------- | ---------------------------------------------------- |
| `--endpoint <https://...>`  | The S3 endpoint; must be `https://`. Required.       |
| `--bucket <name>`           | Required                                             |
| `--region <r>`              | The signing region, such as `auto` for Tigris and R2 |
| `--prefix <p>`              | Prepended to every key as is, such as `rrun/`        |
| `--access-key-id <id>`      | With `--secret-access-key`; else `AWS_ACCESS_KEY_ID` |
| `--secret-access-key <key>` | With `--access-key-id`; else `AWS_SECRET_ACCESS_KEY` |

Any S3-compatible store works: AWS S3, Tigris, R2, or a neocloud's. The key
needs to read and write objects under the prefix. rrun addresses the bucket
path-style (`<endpoint>/<bucket>/<key>`), seals the key, and shows back only the
endpoint, bucket and prefix. It does not test the bucket when you set it; after
the next job ends, its objects should be there.

## What is stored

Per job, under `<prefix>orgs/<organization id>/jobs/<job id>/` (the prefix is
prepended as is, so end it with `/`):

- `logs/<first seq>-<last seq>.jsonl.gz`: one chunk of lines, each
  `{"seq","at","stream","text"}`, with 12-digit zero-padded sequence numbers.
- `telemetry/<first at>-<last at>.jsonl.gz`: one chunk of telemetry samples,
  about 15 minutes of them, named by the first and last sample's time in epoch
  milliseconds, zero-padded to 13 digits so the keys sort by time. Each sample
  is `{"at","cpuPercent","ramUsedGb","ramTotalGb","gpus":[...]}`.
- `telemetry.jsonl.gz`: all of a job's samples in one file, for jobs that ended
  before telemetry was chunked.

The files are plain gzipped JSON Lines, readable with your own tools.

## Changing storage

A change applies at once, to running jobs too: their next flush goes to the new
store. Chunks already written stay where they are, and rrun reads a chunk only
from the store the organization uses now; a read of an older chunk from the
previous store answers with one `[rrun]` line saying those lines are no longer
readable, and telemetry from the previous store is left out. The objects
themselves are untouched; read them from the old bucket directly.

Bundles (the packed folders) always stay in rrun's own bucket, with its 30-day
retention. Changing storage needs a signed-in person, not an API key.

---

rrun by 041 documentation. Every page: https://rrun.041.io/llms.txt