Creating and modifying Madrigal experiments

Adding data to Madrigal involves two steps. The first step is creating Madrigal data files, as discussed in the previous section. The next is installing them on the Madrigal database, with the appropriate metadata and any additional plots or other information you want to supply.

There are a number of scripts supplied with Madrigal to simplify the task of creating or modifying experiments on Madrigal. Use these scripts if you want to:

Create an experiment using a pre-existing Madrigal file
Create an experiment when no Madrigal file yet exists (such as to create a real-time file)
Modify the status of a given experiment
Add a new file to an experiment
Modify an existing file in an experiment
Change the status of a file in an experiment
Remove a file from an experiment
Adding auxiliary plots and information to a Madrigal experiment

After changing a Madrigal experiment using one or more of the scripts above, the script madroot/bin/updateMaster must be run for the changes to take effect. That's because these scripts change the metadata files in the local directory, and updateMaster combines all the local metadata files into the ones in madroot/metadata, which the used by the user interface to Madrigal.

This page also describes how you can add auxiliary information to a given Madrigal experiment.

Create an experiment using a pre-existing Madrigal file

If you have already created a Madrigal file for a given experiment, and want to add that new experiment to Madrigal, use the script createExpWithFile.py, located in madroot/bin. Usage:

createExpWithFile.py is a script used to create a new Madrigal experiment
based on an already existing file. Information such as the duration of the
experiment is obtained by analyzing the file.

With Madrigal 3, accepts either old CEDAR database format files, or CEDAR Madrigal
Hdf5 files.

Required arguments:

--madFilename - full path to the complete Madrigal file. Basename will
be maintained. If old CEDAR databse format, .hdf5 will
be appended

--expTitle - experiment title. Use quotes if title contains spaces.

--permission - 0 for public, 1 for private (restricted to certain IP range)
(both the experiment and the file will set)

--fileDesc - file decription

Optional arguments:

--instCode - instrument code. If this argument missing, instrument code is
taken from file, but error is thrown if more than one kinst found.

--category - 1=default, 2=variant, or 3=history If this argument is missing,
1 (default) used.

--dirName - directory name to use for experiment. If not given, the directory
name will be the default name DDmmmYY[optChar]. Cannot contain "/"

--optChar - optional character to be added to experiment directory if no dirName
given. If dirName argument given, this argument ignored. optChar
is used if the default directory name DDmmmYY is used for
more than one experiment created for a given instrument on a given day.
For example, if --optChar=h for a MLH experiment on September 12, 2005,
then the experiment directory created would be experiments/2005/mlh/12sep05h.

--kindat - Set file kindat independently from one (or more) in file

--experimentsDirNum - the number to be appended to the experiments directory, if experiments
directory being used is of the form experiments[0-9]* instead of just
experiments. For example, if experimentsDirNum is 7, then the experiment
would be created in MADROOT/experiments7 instead of MADROOT/experiments.
Default is to create in experiments directory.

--PI - set Principal Investigator for this experiment

--PIEmail - set PI email for this experiment

--fileAnalyst - set file analyst name for this file. This will default to blank.

--fileAnalystEmail - set file analyst email for this file. This will default to blank.

Example: If you have already created the Madrigal file mlh060120g.001.hdf5 in the /tmp directory, and the experiment name is Calibration, the file should be public, and the description is "Final", you would enter:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.001.hdf5 \
    --expTitle="Calibration" --permission=0 --fileDesc="Final"

If your input file is an old style CEDAR database file mlh060120g.001, the only difference is that there is not hdf5 extension:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.001 \ 
    --expTitle="Calibration" --permission=0 --fileDesc="Final"

Create an experiment when no Madrigal file yet exists

If you want to create an experiment for a Madrigal data yet to be created (such as when you want to create the Madrigal file in real-time), use the script createRTExp.py, located in madroot/bin. Usage:

createRTExpWithFile.py is a script used to create a new Madrigal experiment
   that will contain real-time files.  These real-time files are assumed not to exist
   yet.

Required arguments::

 --startDate - experiment start date in form YYYY-MM-DD

 --inst - instrument code or 3-letter Madrigal mnenonic

 --expTitle - experiment title. Use quotes if title contains spaces.

 --rtFiles - comma-separated list of realtime file basenames to be created

 --kindats - comma-separated list of ints or single int of kindats for each realtime file.
   The length and order must be the same as rtFiles.  If only one
   given, it is assumed that all rtFiles have the same kindat.

 --fileDescs - comma-separated list of file descriptions. If the file description contains spaces,
   quotes must be used.

Optional argument::

 --numDays - number of days the experiment is estimated to run - if not given, and no start
   and end times specified, defaults to one day.  Error raised if endDate and endTime
   also specified.

 --startTime - start time in form HH:MM:DD.  Defaults to 00:00:00

 --endDate - end day in form YYYY-MM-DD. endTime must also be specified

 --endTime - end time in form HH:MM:DD.  endDate must also be specified

 --permissions - comma-separated list of 0 for public, 1 for private (restricted to certain IP range).
   If only one given, it is assumed it applied to all.  If this argument is not
   given, it defaults to 0 (public)

 --dirName  - directory name to use for experiment.  If not given, the directory
   name will be the default name DDmmmYY[optChar].  Cannot contain "/"

 --optChar  - optional character to be added to experiment directory if no dirName
   given.  If dirName argument given, this argument ignored.  optChar
   is used if the default directory name DDmmmYY is used for
   more than one experiment created for a given instrument on a given day.
   For example, if --optChar=h for a MLH experiment on September 12, 2005,
   then the experiment directory created would be experiments/2005/mlh/12sep05h.

 --security - overall experiment access.  0 for public, 1 for private, -1 for ignore.
 Defaults to public (0)

--experimentsDirNum - the number to be appended to the experiments directory, if experiments
                      directory being used is of the form experiments[0-9]* instead of just
                      experiments.  For example, if experimentsDirNum is 7, then the experiment
                      would be created in /Users/brideout/madroot31/experiments7 instead of MADROOT/experiments.
                      Default is to create in experiments directory.

 --PI - set Principal Investigator for this experiment

 --PIEmail - set PI email for this experiment

 --fileAnalyst - set file analyst name for these files (no commas allowed).  
        This will default to blank.  If a different
        fileAnalyst is desired for each file, the names must be comma separated.

 --fileAnalystEmail - set file analyst email for this file.  This will default to blank.
        If a different fileAnalyst email is desired for each file, the emails must be comma separated.

Example: If you plan to create two real-time Madrigal files called mlh021001a.000.hdf5 and mlh021001b.000.hdf5 for the Millstone Hill Radar for an experiment planned to run 2 days, with both files having the kindat 3410, you would enter:

/opt/madrigal/bin/createRTExp.py --startDate=2002-10-01 --numDays=2 --inst=mlh --expTitle="test experiment" \
    --rtFiles=mlh021001a.000.hdf5,mlh021001b.000.hdf5 --kindats=3410 \
    --fileDescs="preliminary - single pulse,preliminary - alternating code"

Modify the status of a given experiment

All of the experiment attributes can be changed by the script changeExpStatus.py, located in madroot/bin. The most common reason you'd want to run this script is to change an experiment's security, with the options being 0 (public), 1 (limited by IP address), and -1 (ignored, or hidden from everyone). To completely remove an experiment, simply delete the directory. However, setting security to -1 allows you to bring the experiment back at some later time by running changeExpStatus.py again.

A number of other attributes can also be modified, but most are set automatically, and should not need modification.

Usage:

changeExpStatus.py is a script used to change the status of an
 existing Madrigal experiment.  The following attributes can be changed:

 expUrl
   experiment name
   siteID
   start date
   start time
   end date
   end time
   instrument code
   security (public, private. ignore)
   PI (optional - overrides default for instrument)
   PIEmail (optional - overrides default for instrument

Required argument:

 --expDir - full path to experiment directory. Example:
   "/opt/madrigal/experiments/1998/mlh/20jan98"

Optional arguments - set these to change an experiment attribute:

 --expUrl - must be in form <cgi base>/madtoc/YYYY/<3 letter lower case inst code>/<expDir>
   example: http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97g

 --expName - experiment name.  Quotes required if contains spaces.  Example: "World Day"

 --siteID - Madrigal siteID of where data will be stored.  Error raised if not the siteID
   of the local Madrigal site. Example: 4

 --startDate - new start date of experiment (UT).  In form YYYY-MM-DD.  Example: 1998-01-20

 --startTime - new start time of experiment (UT).  In form HH:MM:DD.  Example: 12:30:00

 --endDate - new end date of experiment (UT).  In form YYYY-MM-DD.  Example: 1998-01-21

 --endTime - new end time of experiment (UT).  In form HH:MM:DD.  Example: 23:30:00

 --inst - new instrument code.  Example: 30

 --security - new security code.  Allowed values are 0 for public, 1 for private (limited IP range access)
                -1 for ignore, 2 for archived experiment, 3 for private (limited IP range access) archived
                experiment.

 --PI - set Principal Investigator for this experiment

 --PIEmail - set PI email for this experiment

Example: to change to experiment /opt/madrigal/experiments/2006/mlh/20jan to be private, you would run:

/opt/madrigal/bin/changeExpStatus.py --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --security=1

Add a new file to an experiment

If you want to add a new file to an existing Madrigal experiment, use addFileToExp.py, located in madroot/bin. You can use this script to add a completely new file, or to update an existing one. It you update an existing file and want to change the status of the older file to history or variant, use the script changeFileStatus.py.

Usage:

addFileToExp.py is a script used to add a new Madrigal file to an
existing experiment.  Information such as the duration of the
experiment is updated by analyzing the file.

With Madrigal 3, accepts either old CEDAR database format files, or CEDAR Madrigal 
Hdf5 files.

Required arguments:

    --madFilename - full path to the complete Madrigal file. Basename will
                    be maintained. If old CEDAR databse format, .hdf5 will 
                    be appended

    --expDir - full path to experiment directory. Example:
               "/opt/madrigal/experiments/1998/mlh/20jan98"
    
    --permission - 0 for public, 1 for private (restricted to certain IP range)

    --fileDesc - file decription

Optional arguments:

    --category - 1=default, 2=variant, or 3=history If this argument is missing,
                 1 (default) used.

    --kindat  -  Set file kindat independently from one (or more) in file
    
    --skipNotify - if this flag set (no arguments), registered users will NOT be notified
                   of this change.  The default is to email all registered users.
                   
    --fileAnalyst - set file analyst name for this file.  This will default to blank.
    
    --fileAnalystEmail - set file analyst email for this file.  This will default to blank.

Example: If you want to add the new file /tmp/mlh060120g.002.hdf to an existing experiment, you would enter:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.002.hdf5 \
     --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --permission=0 \
     --fileDesc="alternative analysis"  --category=1

If you are adding an old style Cedar databse file /tmp/mlh060120g.002 to an existing experiment, you would enter the same command except without the .hdf5 extension:

/opt/madrigal/bin/createExpWithFile.py --madFilename=/tmp/mlh060120g.002 \
     --expDir=/opt/madrigal/experiments/2006/mlh/20jan06 --permission=0 \
     --fileDesc="alternative analysis"  --category=1

Modify an existing file in an experiment

If you want to modify an existing file in a Madrigal experiment, use updateFileInExp.py, located in madroot/bin. Note that if the modification in the file is significant, it is preferable to make the old file a history file using changeFileStatus.py, and to add a new file with a different name using addFileToExp.py.

Usage:

updateFileInExp.py is a script used to update an existing Madrigal file in an
   existing experiment. Information such as the duration of the
   experiment is updated by analyzing the file. This script is use to replace
   an existing Madrigal file. Use addFileToExp.py to add a new file, and
 changeFileStatus.py to change any file attribute.

Required arguments:

 --madFilename - full path to the new version of the Madrigal file. Basename will
   be maintained.

 --expDir - full path to experiment directory. Example:
 "/opt/madrigal/experiments/1998/mlh/20jan98"

Example: To modify the existing file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5 with the file /tmp/mlh021001a.000.hdf5, you would enter:

/opt/madrigal/bin/updateFileInExp.py --madFilename=/tmp/mlh021001a.000.hdf5 \
 --expDir=/opt/madrigal/experiments/2002/01oct02

Change the status of a file in an experiment

If you want to change the status of any existing file in an experiment, such as to make a default file into a history file, use changeFileStatus.py, located in madroot/bin. Usage:

changeFileStatus.py is a script used to change the status of an
existing Madrigal file.  The file permission, the file description,
or the file category can be changed.

Required arguments:

    --filename - basename of existing Madrigal file. 

    --expDir - full path to experiment directory. Example:
               "/opt/madrigal/experiments/1998/mlh/20jan98"

Optional arguments - set these to change a file attribute:
    
    --permission - 0 for public, 1 for private (restricted to certain IP range)

    --fileDesc - file decription

    --category - 1=default, 2=variant, 3=history, or 4=realtime
    
    --fileAnalyst - set file analyst name for this file.  
    
    --fileAnalystEmail - set file analyst email for this file.

Example: If you want to change the status of /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5 to be a history file, you would enter:

/opt/madrigal/bin/changeFileStatus.py --filename=mlh021001a.000.hdf5 \
    --expDir=/opt/madrigal/experiments/2002/01oct02 \
    --category=3

Remove a file from an experiment

To completely remove a file from an existing experiment, rather than simply make it a history file, use removeFileFromExp.py, located in madroot/bin. Usage:

removeFileFromExp.py is a script used to remove an existing Madrigal file from an
existing experiment. Information such as the duration of the
 experiment is updated by analyzing the remaining files.

Required arguments:

 --filename - basename of the Madrigal file to be removed.

 --expDir - full path to experiment directory. Example:
 "/opt/madrigal/experiments/1998/mlh/20jan98"

Example: To remove the file /opt/madrigal/experiments/2002/01oct02/mlh021001a.000.hdf5, you would enter:

/opt/madrigal/bin/removeFileFromExp.py --filename=mlh021001a.000.hdf5 \
     --expDir=/opt/madrigal/experiments/2002/01oct02

Adding auxiliary plots and information to a Madrigal experiment

Since Madrigal is a web-based application, you can also display auxiliary plots and information about your experiment as web pages. If you add additional web documents according to the rules below, these documents will also show up through the standard Madrigal interface.

Subdirectories containing an html file named index.html . Links to these files will show up the experiment page. The page title will be displayed as a link.
Html pages in the main directory, and again links to these files will show up the experiment page. The page title will be displayed as a link.
A file called notes.txt in the main directory, displayed as notes.txt.
Plots or other files related to individual records in the dataset. Links to these files appear next to the appropriate record in the summarizeCedarFile.cgi page. These files must be located in the subdirectory "plots/[file name]/records" under the experiment directory, where "file name" is the base name of the experiment record.. In order for the script to determine which file goes with which record, the files must include a five digit number somewhere in its name. For example, plot00027.gif would appear as a link next to record 27 in summarizeCedarFile.cgi. More than one file can be associated with a given record.
- If you want to create individual record plots for an incoherent scatter radar with the standard parameters Te, Ti, Ne, and ion velocity, simply run the script createRecordPlots.py. For example, to create all the individual record plots for the Madrigal file mlh980120g.001.hdf5, you would run a command similar to the following:
```
/opt/madrigal/bin/createRecordPlots.py /opt/madrigal/experiments/1998/mlh/20jan98/mil980120g.001.hdf5
```

As an example of plots associated with individual records, lets say you have a Madrigal experiment in the directory /opt/madrigal/experiments/2002/01oct02, with a file named mlh021001a.001.hdf5. If this file had 10 records, you could then create 10 plot files called plot001.png through plot010.png. You would then create the subdirectories plots/mlh021001a.001.hdf5/ under the main directory /opt/madrigal/experiments/2002/01oct02/, and put those 10 plot files there.