How Madrigal data is organized

In the Madrigal database, the data are organized by experiment. An experiment consists of data from a single instrument, and, as a rule, is meant to address a particular scientific goal. Most often an experiment will correspond to a particular set of operating modes run for a contiguous interval of time. For example, the data collected by an incoherent scatter radar during a world day would constitute an experiment. More complicated situations exist, and judgment may have to be exercised in determining what constitutes an experiment. For example, an experiment might be interrupted temporarily in order to use a different set of operating modes in support of a satellite overpass. In this case there could be two experiments which overlap in time.

To each experiment there corresponds a directory. These directories are of the form madroot/experiments[0-9]*/<year>/<instrument>/<directory>, where year is the four-digit year. The next level is the experiments[0-9]*, where the directory must begin "experiments" and then may be followed by any number of digits. This is to allow additional hard drives to be mounted to expand the directory space. The next level, instrument, is the 3-letter mnemonic for the instrument (set in the instTab.txt file), and <directory> is an arbitrary directory name. An example might be /opt/madrigal/experiments/1997/son/06jan1997_001, which would contain a 1997 Sondrestrom (son) experiment. The use of the starting date in the experiment directory name is not required.

Prior to Madrigal 2.5, the final directory name was required to be in the form DDmmmYY*, where the date was the start date of the experiment and * is an optional character to distinguish different experiments with the same start date. For example, /opt/madrigal/experiments/1997/son/06jan97 contained Sondrestrom data for an experiment beginning on 6 January, 1997, where year =1997, instrument =son, start_date =06jan97, and there was no optional character because there was only one Sondrestrom experiment starting on that date. This convention has been dropped as a requirement, although it is still commonly used as a convention.

Previously, experiments that did not follow the above naming convention were effectively "hidden". With Madrigal 2.5, the security field in the metadata file expTab.txt can be used to hide (or restrict access to) an experiment. (See the change experiment status tool).

Each experiment directory must contain:

The Experiment Table entry (expTab.txt) for this experiment. This file is created automatically if you use the tools described in the creating experiments section.

Each experiment directory may contain:

The metadata file fileTab.txt for this experiment. This table must have one entry for each Madrigal format file in the experiment. This file is created automatically if you use the tools described in the creating experiments section.
One or more datasets in CEDAR Madrigal Hdf5 format.
Subdirectories containing an html file named index.html . Links to these files will show up the experiment page.
Html pages in the main directory, and again links to these files will show up the experiment page.
Plots or other files related to individual records in the dataset. Links to these files appear next to the appropriate record in the summarizeCedarFile.cgi page. These files must be located in the subdirectory "plots/[file name]/records" under the experiment directory. In order for the script to determine which file goes with which record, the files must include a five digit number somewhere in its name. For example, plot00027.gif would appear as a link next to record 27 in summarizeCedarFile.cgi. More than one file can be associated with a given record.

An essential role for a Madrigal administrator is to run the script madroot/bin/updateMaster on a regular basis. This script performs a number of functions:

It gathers all the separate metadata files in the individual experiment directories into the central directory madroot/metadata.
It gathers metadata files from other Madrigal sites into the central directory madroot/metadata.
It updates geophysical files from the OpenMadrigal site.

It is recommended that this script be set as a cron job to run once a day.

For Madrigal sites with limited bandwidth, updateMaster can occasionally be run with the -s flag. This will skip any possible download of the latest geophysical files, and so maybe run faster. However, it is still important to run updateMaster without the -s flag on a regular basis.

A complete description of the various ways to add and modify experiments in a Madrigal database is given in the next section. The updateMaster script must be run after each change described in that section for the changes to take effect.

Note that an experiment need not contain any Madrigal files.