Archive Organization

Our open archive is organized in a manner that enables easy browsing. Datasets are organized into directories in a YYYY/MM/DD style of tree, with many datasets also including additional levels for location and hour. Universally across all available data, timestamps are represenated in UTC time. All filenames are unique, regardless of their tree depth. These standards enable easy human-readability and also programmatic retrieval.

The archive is organized into two separate top-level trees: by project and by instrument type.

Within the instrument array trees (ie. themis/asi, trex/rgb, etc.), you will find several datasets. Types of datasets are organized into their own trees, depicted by 'streamX' or 'lX' (for 'level'). Older datasets will use the 'streamX' format, and newer datasets will use the 'lX' format. Below is a breakdown of what each means:

Stream format:

• stream0: full resolution raw data
• stream1: real-time data streams
• stream2: summary-level data products (keograms, montages, etc.)
• stream3: ancillary files, usually not needed for analysis
• grid_files: network-wide grid files

Level format:

• l0: full resolution raw data
• l1: full resolution processed data
• l2: summary-level data products (keograms, montages, etc.)
• l3: high level data products (network-wide grid files)

Additional:

• rt-fullres: real-time live feed images and movies
• skymaps: UCalgary-designed skymapping files (used to project all-sky imager data onto maps)

Note that some datasets do not follow either stream/level formats, however the organization will be easy to understand. All datasets have consistent naming and organization standard, and will not change partway through the dataset.