Arcus

Arcus is a storage system which can be accessed both as a traditional file system, and with the S3 API. The file system is accessible from the Stratus cluster under /arcus. The S3 API is accessible at https://arcus-s3.nsc.liu.se/. S3 access is read-only.

S3 examples

Downloading an object (file) with AWS Command Line Interface (CLI):

aws s3 --endpoint-url=https://arcus-s3.nsc.liu.se cp s3://example/2022/01/28/00/example.grib2 .

Listing all objects (files) under a prefix (directory) with AWS Command Line Interface (CLI):

aws s3 --endpoint-url=https://arcus-s3.nsc.liu.se ls s3://example/2022/02/28/00/

Downloading an object (file) with S3cmd:

s3cmd --host=https://arcus-s3.nsc.liu.se --host-bucket= get s3://example/2022/01/28/00/example.grib2

Listing all objects (files) under a prefix (directory) with S3cmd:

s3cmd --host=https://arcus-s3.nsc.liu.se --host-bucket= ls s3://example/2022/01/28/00/

If the client you are using for S3 requires you to specify a region, then specify us-east-1.

The file system and S3

The directory /arcus/metcoop is connected to the S3 API. For instance, the file /arcus/metcoop/example/2022/01/28/00/example.grib2 is accessible as the object s3://example/2022/01/28/00/example.grib2 with S3.

Every directory at the top of /arcus/metcoop is represented as a bucket in S3. Objects in S3 must reside in a bucket, and therefore regular files at the top of /arcus/metcoop cannot be accessed with S3.

Empty directories in the file system cannot be represented in S3 and are therefore not visible with S3.

After adding new files to the file system, there is a delay before they are visible as objects with S3. The delay varies and depends on how large the new files are, but can generally be expected to be around 10 minutes.

Backups

Arcus is backed up to tape at least weekly. If you need something restored from a backup, please contact smhi-support@nsc.liu.se. Sometimes you can also restore deleted files yourself by browsing the file system snapshots.

Snapshots

The /arcus file system uses snapshots. A snapshot is a read-only view of some previous point in time on the file system. Snapshots can be used to restore deleted files.

You will find the snapshots in the directory /arcus/.snapshots. In the snapshot directory, snapshots are named in the format YYYY-MM-DDTHHMM+0000. The suffix +0000 indicates that the time is in UTC. The oldest snapshot is usually 2 days old.

Note that snapshots are taken on a best-effort basis and are not guaranteed to exist. If the file system runs out of space, snapshots may be deleted.