Appearance
Downloading Data
Our data archive can be accessed using several methods. Below we describe each method, and provide several examples.
Before you begin
Please have a look at the Archive Organization page before downloading data.
HTTP / HTTPS
You can download data using the browser-based file tree accessible using the links below.
Explore Open Archive:
Programmatic access
If you write code to programmatically download files by scraping these pages, consider using Rsync or FTP instead. The web-based tree format may change without notice and break your code.
Rsync
The archive is also accessible using Rsync. Rsync is quite handy for trivially synchronizing directory trees to you machine.
bash
# Connect to the rsync server and list top-level module
#
# NOTE: this command does not download any data, only lists
rsync rsync://data.phys.ucalgary.cabash
# Get an hour of raw data from the THEMIS ASI instrument at Gillam
rsync -av rsync://data.phys.ucalgary.ca/data/sort_by_project/THEMIS/asi/stream0/2024/01/01/gill*/ut06 ./themis_data/bash
# Perform the simple download example, but do not download any
# data. Instead, only simulate what would happen.
#
# NOTE: this is enabled by the -n option
rsync -avn rsync://data.phys.ucalgary.ca/data/sort_by_project/THEMIS/asi/stream0/2024/01/01/gill*/ut06 ./themis_data/bash
# You can also download data and limit the bandwidth. The number
# is expected to be in Kb/s, so the example below will download
# the data and limit your bandwidth usage to 5Mb/s (note this is
# metabits and not megabytes).
#
# Further information can be found by reading the documentation
# for the rsync command (can see easily using 'man rsync')
rsync -avn --bwlimit=5000 rsync://data.phys.ucalgary.ca/data/sort_by_project/THEMIS/asi/stream0/2024/01/01/gill*/ut06 ./themis_data/FTP
The archive is also accessible using FTP. This is an anonymous FTP site, so if you are using an FTP/SFTP client like WinSCP, FileZilla, or Total Commander, then use the username anonymous and password anonymous.
FTP server: ftp://data.phys.ucalgary.ca
lftp
For efficient recursive downloading from an FTP server, lftp is a great CLI tool with powerful mirroring capabilities.
Below are a few examples for mirroring an hour of data from the THEMIS ASI at Gillam.
bash
lftp -e "mirror --verbose sort_by_project/THEMIS/asi/stream0/2024/01/01/gill_themis19/ut06/ ./themis_data/; quit" data.phys.ucalgary.cabash
lftp -e "mirror --verbose --parallel=4 sort_by_project/THEMIS/asi/stream0/2024/01/01/gill_themis19/ut06/ ./themis_data/; quit" data.phys.ucalgary.caPython and IDL Libraries
Our Python and IDL libraries provide a convenient way to download and work with our data programmatically. They handle data discovery, downloading, reading, and analysis support automatically. See the Software Packages page for details, and the crib sheets (Python, IDL) for detailed examples.
API
You can also access data URLs programmatically via our RESTful API's "Data Distribution" endpoints. See the API page for details and examples.