rrun
All pages
Docs · SetupMarkdown

Log storage

rrun's built-in storage keeps job logs, telemetry and uploaded folders for 30 days, then deletes them. To keep logs and telemetry longer, set your own S3-compatible bucket with o41-rrun storage set or the web app's Storage page; rrun never cleans it up.

o41-rrun storage          # where logs go now, and for how long
o41-rrun storage set --endpoint https://t3.storage.dev --bucket logs --region auto --prefix rrun/
o41-rrun storage rm       # back to rrun's bucket

A job's log lines are buffered in the job and flushed as gzipped JSON Lines chunks: at 1 MiB, every 30 seconds, and when the job ends. Telemetry flushes the same way in chunks of about 15 minutes. The newest log lines and the last two hours of telemetry also stay with the job, so tails and recent charts are fast.

Retention

Where What Kept
rrun's own bucket (the default) Logs and telemetry 30 days after each chunk is written
Your own bucket Logs and telemetry As long as your bucket keeps them
rrun's own bucket, always Uploaded folders (bundles) 30 days after the upload

rrun never deletes from your bucket; set a lifecycle rule there if you want one.

Reading past the retention is not an error: o41-rrun logs, o41-rrun telemetry and the job's page show what is left and say that older data was deleted. A long job's newest log lines and its last two hours of telemetry stay with the job, so they outlive the chunks.

From a week before its 30 days are up, an uploaded folder counts as not uploaded, so the next o41-rrun run uploads it again and its 30 days start over. A job that waits or restarts past its folder's 30 days fails with a message saying the bundle expired; run it again to upload the folder.

Your own bucket

storage set takes:

Flag Meaning
--endpoint <https://...> The S3 endpoint; must be https://. Required.
--bucket <name> Required
--region <r> The signing region, such as auto for Tigris and R2
--prefix <p> Prepended to every key as is, such as rrun/
--access-key-id <id> With --secret-access-key; else AWS_ACCESS_KEY_ID
--secret-access-key <key> With --access-key-id; else AWS_SECRET_ACCESS_KEY

Any S3-compatible store works: AWS S3, Tigris, R2, or a neocloud's. The key needs to read and write objects under the prefix. rrun addresses the bucket path-style (<endpoint>/<bucket>/<key>), seals the key, and shows back only the endpoint, bucket and prefix. It does not test the bucket when you set it; after the next job ends, its objects should be there.

What is stored

Per job, under <prefix>orgs/<organization id>/jobs/<job id>/ (the prefix is prepended as is, so end it with /):

  • logs/<first seq>-<last seq>.jsonl.gz: one chunk of lines, each {"seq","at","stream","text"}, with 12-digit zero-padded sequence numbers.
  • telemetry/<first at>-<last at>.jsonl.gz: one chunk of telemetry samples, about 15 minutes of them, named by the first and last sample's time in epoch milliseconds, zero-padded to 13 digits so the keys sort by time. Each sample is {"at","cpuPercent","ramUsedGb","ramTotalGb","gpus":[...]}.
  • telemetry.jsonl.gz: all of a job's samples in one file, for jobs that ended before telemetry was chunked.

The files are plain gzipped JSON Lines, readable with your own tools.

Changing storage

A change applies at once, to running jobs too: their next flush goes to the new store. Chunks already written stay where they are, and rrun reads a chunk only from the store the organization uses now; a read of an older chunk from the previous store answers with one [rrun] line saying those lines are no longer readable, and telemetry from the previous store is left out. The objects themselves are untouched; read them from the old bucket directly.

Bundles (the packed folders) always stay in rrun's own bucket, with its 30-day retention. Changing storage needs a signed-in person, not an API key.