This document is meant as a manual for administering the OpenMadrigal web site. It contains information on both the developer's site http://cedar.openmadrigal.org/openmadrigal and the archive site cedar.openmadrigal.org. It also contains information on releasing Madrigal and its associated client API's.
The cedar.openmadrigal.org site serves as a central archive of all Madrigal sites. It also serves as a home for instrument data not maintained by local Madrigal sites.
All data is downloaded from remote sites using either the script import_madrigal.py for Madrigal 3 sites, or import_mad2_mad3.py to import from old Madrigal 2 sites. These script may be found in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts. They should be set up as cron jobs to download remote Madrigal data on a regular basis, according to agreements with remote Madrigal site administrators. All data downloaded is logged in /opt/madrigal3/metadata/userdata/<site>_import_<year>.txt. See Madrigal crontab section below for a description of all cron jobs running on Madrigal.
Administrators can access the log files particular to their instruments using the script http://cedar.openmadrigal.org/getLogAdmin.py. As the CEDAR Madrigal database administrator, there are two tasks to support this. The first is the creation of passwords. Passwords can be created using the script /opt/madrigal3/bin/setAdminPassword.py with usage setAdminPassword.py <username> <password>. This script can also be used to modify existing passwords. This script is located in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts. The username is either the site name as set in $MADROOT/metadata/siteTab.txt, or set in $MADROOT/metadata/hostSite.txt. The second task is to maintain the file hostSite.txt. This is a metadata file unique to CEDAR Madrigal, and defines virtual sites for PI's who host their data directly on CEDAR Madrigal. It is kept in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/CEDAR_site. The columns of this file are:
<fake site id>, <fake site name>, <kinst 1> [,<kinst 2> ...]
Example:
10001,MillstoneHighResFP,5360
Whenever a new PI adds a new instrument to be hosted at CEDAR Madrigal, this file must be updated, and a new username/password pair created using setAdminPassword.py. PI's with multiple instruments can have more than one kinst code (comma separated) on the line. The user name is the <fake site name>, so either create it without spaces, or use quotes when running setAdminPassword.py. Send the PI the site name and password after you have tested it.
Local Madrigal administrators are free to modify the kind of data (kindat) descriptions of their data at any time by modifying the local file $MADROOT/metadata/typeTab.txt. The Cedar archive site needs a central repository of these kindat descriptions. To avoid problems when multiple Madrigal sites use the same kindat codes, Madrigal now allows kindat descriptions to be indexed by a combination of instrument code and kindat code. With Madrigal 3.0, the typeTab.txt file will be able to have entries of the form <kinst>_<kindat>,<description>, in addition to the present format of <kindat>,<description>. This feature is already installed on the Cedar archive Madrigal server. There is also a cron job checkCedarTypeTab.py that checks all remote Madrigal sites for any updates to their typeTab.txt file, and imports then into the Cedar archive site's typeTab.txt file. This script can be found in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts.
The script changeInstAccess.py can be useful in administering this site. It allows you to change the status of all experiments with a given instrument kinst with one command. This script may be found in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts.
The OpenMadrigal developer's web site serves a number of purposes:
This manual also covers the building and testing of a new Madrigal release.
Two cron jobs are required to collect these data, createExpUpdate.py and createGeoUpdate.py.
createExpUpdate.py - located in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts. Usage: createExpUpdate.py [-e emailAddress] [-e emailAddress2] ... where any number of email addresses can be notified of the results. This script collects the metadata files expTab.txt and fileTab.txt from each of the Madrigal sites listed in siteTab.txt, and stores them in the directory metadata/[site_name]_[site_id]. The site_name and site_id are also set by siteTab.txt. This cron job should be set to run at least once a day, and the results should be emailed to the OpenMadrigal administrator. The results are accessed via the web from the url http://cedar.openmadrigal.org/static/distributionFiles/metadata.
createGeoUpdate.py - located in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/scripts. Usage: createGeoUpdate.py [-e emailAddress] [-e emailAddress2] ... where any number of email addresses can be notified of the results.. This script will update the geofil.tar.Z and ig_rz.dat file on OpenMadrigal. Also updates the file status.dat which indicates the last update date of each type of geophysical data. All the files updated are accessed via the url http://cedar.openmadrigal.org/static/distributionFiles/. Due to relatively infrequent updates to the base files, this cron job is now run only once a week.
Three core metadata files are also distributed via updateMaster:
The OpenMadrigal administrator is responsible for updating these three files in Subversion at https://apollo.haystack.mit.edu/svn/OpenMadrigal/trunk/madroot/metadata (or metadata3 for siteTab.txt) whenever they are changed.
The script checkMetadata.py can be set up as a cron job to monitor whether all Madrigal sites have these three up-to-date metadata files. This script is located in Subversion under OpenMadrigalSVN/trunk/madroot/source/madpy/scripts/bin.
Finally, this capability requires that the cedar app compareToArchive.py be installed on the Cedar server at url http://cedar.openmadrigal.org/compareToArchive.py. This django app can be found in Subversion under OpenMadrigalSVN/trunk/madroot/source/madpy/djangoMad/apps/cedar.
The latest version of the crontab.txt file used by midasop on the Madrigal machine at Haystack (which includes both the CEDAR and the Millstone Madrigal installations) can be found in Subversion under MillstoneSVN/instruments/millstone_uhf_radar/millstone_madrigal/trunk/OpenMadrigal/crontab.txt. All scripts in this crontab are in Subversion in the OpenMadrigal repository. This script includes both administrative jobs (such as updateMaster, import_madrigal.py, etc) and jobs that automatically load data into the CEDAR Madrigal site from instrument PI's who do not run their own local Madrigal sites.
Madrigal has client API's written in python, Matlab, and IDL. All three have testMadrigal* scripts to run that test all functionality. They are rebuilt whenever you run make -f Makefile.gnu in the main development directory. Version with *.tar.gz and *.zip are built for each of the three. To release new version, edit the file in Subversion OpenMadrigalSVN/trunk/madroot/source/madpy/djangoMad/apps/cedar/templates/madDownload.html with the new release number. This madDownload.html needs to be installed in /opt/madrigal3/source/madpy/djangoMad/apps/cedar/templates. The new source files should be installed on madrigal in /opt/madrigal3/source/madpy/djangoMad/madweb/static/distributionFiles/. In addition, the version of madrigalWeb on pip needs to be updated, using: python -m twine upload dist/*
The Madrigal derivation engine contains a number of models that are under active development, and so should be updated if possible with each new Madrigal release. These models are listed below.
All Fortran code that uses reals is converted to use all doubles using nag_apt tools.
From Subversion at Millstone, checkout OpenMadrigalSVN/trunk/madroot. The file source/configure.ac has the line AC_INIT([Madrigal] that should be updated with the Madrigal version. Since autotools is used in building a distribution, you want to be sure that your machine has up-to-date autotools. For Madrigal 2.5, I used autoconf 2.6.3, automake 1.10.2, libtool 2.2.4, and m4 1.4.12. To create the needed autotool files, cd to source and run:
To make the distribution file, cd to madroot, and run "make -f Makefile.gnu". The file MANIFEST in the madroot directory controls which files are included in the tar file. The general test procedure is to build the distribution, and then test install on any available unix machines, both as updates and as new installs. There are also regression tests to be run on the core C and Fortran libraries, and on the Python library.
The release number is set in source/configure.ac and in source/madpy/setup.py. Also the files madContents.html and whatsNew.html need to be updated to describe the new release.
The python regression tests are in Subversion in OpenMadrigalSVN/trunk/madroot/source/madpy/scripts/regressionTests. Each of the following three regression tests are run as $MADROOT/bin/python script:
Make sure pdoc is installed in $MADROOT/bin/python.
Run pygmentize. You will need to copy in the style code in the head, and the body code.
Revised: Aug 15, 2019