Previous: Brief History   Up: Doc home   Next: Madrigal user's guide

What's new in Madrigal 3.0?

Madrigal 3.0 Release - Sept 2017

Migration to Hdf5 file format

With Madrigal 3.0, the old 16 bit integer based file format has been replaced with Hdf5. However, all the data model and well-defined parameters have been retained. The CEDAR Hdf5 format is designed to be completely self-contained. That is, any user with an Hdf5 reader can fully understand all data in the file without any reference to documentation. A full description of the format can be found here.

Web interface generates scripts to download any amount of data

With Madrigal 3, you can generate a script command to download a whole series of files with a few click. You can also create a script that will filter those same files and allow you to choose the parameters you want, again with just a few clicks. These scripts can be run with python, Matlab, or IDL.

Download files in a new format

Prior to Madrigal 3, files could be downloaded only as ascii or Hdf5 (or the difficult to understand CEDAR format). Now they can be downloaded as ascii, Hdf5, or netCDF4.

Get data with selected/derived parameters and filters in new formats

Prior to Madrigal 3, data with selected/derived parameters and filters was available only as ascii. Now it can be downloaded as ascii, Hdf5, or netCDF4.

Independent spatial parameters now built in to data model

Prior to Madrigal 3.0, there was no way to automatically tell what the independent spatial parameters were in vector data. With Madrigal 3.0, any data with vector data must define its independent spatial parameters, which can be found in the Hdf5 Metadata group. This allows Madrigal to automatically add array layouts of the data, making for easier plotting.

All new web interface

A much simplified web interface based on Django and bootstrap. Developed with much assistance from Jicamarca Observatory and Jose Antonio Sal y Rosas Celi.

Simple FTP-like web interface added

Designed for non-native English speakers. The url follows a very simple pattern that can be easily parsed if the user in unable to understand the Madrigal API's and automatically generated scripts.

Create CEDAR Hdf5 files with either Matlab or python

Prior to Madrigal 3, there was only a python API to easily create Madrigal files. Now both Matlab and python can be used to create CEDAR Hdf5 files.

All metadata easily accessible

The Access metadata menu item on the main navigation menu allows easy access to all Madrigal CEDAR metadata, including Madrigal sites, instruments, parameter definitions, and kind of data codes.

Madrigal Release - March 22, 2012

Added support to file download cgi scripts to allow logging of remote API requests to download files.

Madrigal Release - March 13, 2012

Added popup javascript window to warn users not to click "Download File" button multiple times.

Madrigal 2.6.3 Release - Febuary 27, 2012

Small bug fixes in 3 cgi scripts affecting plots: madExperiment.cgi, madDataDisplay,

Madrigal 2.6.2 Release - Febuary 8, 2012

Small bug fix involving creating Hdf5 export files in

Madrigal 2.6.1 Release - January 20, 2012

Small bug fix involving a memory leak in _Madrec.c and a rare problem creating Hdf5 export files in

Madrigal 2.6 Release - November 2011

New simple local web user interface

A new easy to use interface to access local Madrigal data was developed by Jicamarca with support from Millstone Hill staff. This interface in now very robust and has been tested with data from all Madrigal sites.

HDF5 file download availablity

Users can now download data files as HDF5, in addition to simple column delimited ascii and the more complex CEDAR formats. This is also the result of a Jicamarca/Millstone Hill collaborations.

Users can register interest in experiments or instruments

Users can now register interest in experiments or instruments, and get emails whenever that Madrigal experiment or instrument is updated.

Administrative improvements - can now add external disks to experiments

It is now easy to expand the Madrigal database by mounting additional hard disks to make more room for data.

New metadata - experiment PI's and analyst

Experiments now have direct email links to the experiment PI. These PI's can be updated on an experiment or intrument basis.

Global seach now more robust

The global search UI now generates scripts so you can run the global search right from your computer using either the python remote API, the Matlab remote API, or the IDL remote API.



Madrigal 2.5.2 Release - May 2009

A bug fix release - no new features.

Madrigal 2.5.1 Release - March 2009

This minor revision to Madrigal 2.5 was needed to support using Madrigal as an archiving database. The experiment metadata was expanded to allow archived experiments. Archived experiments are not shared between Madrigal sites, but are visible as local experiments.

Madrigal 2.5 Release - February 2009

Simplification of Web User Interface

Both the Browse for Individual Madrigal Experiments and the Global Madrigal Database Report web interface have been simplified. Searching for instruments under Browse for Individual Madrigal Experiments is now easier through the use of an instrument category selector.

One step file printing available

Under Browse for Individual Madrigal Experiments, users can now choose to print an ascii version of any Madrigal file with one click. With this option they can not include any derived parameters or data filters.

Installation simplified

Autotools is now used to compile all code, significantly reducing the number of parameters in the madrigal.cfg configuration file.

64-bit tested

Madrigal has now been fully tested as a 64-bit application. It is important that all Madrigal installations switch to 64-bit machines by the year 2037, because 32-bit unix cannot handle dates beyond then.

International Reference Ionosphere (IRI) derived parameters now available

All parameters calculated by the International Reference Ionosphere (IRI) model can now be selected as derived parameters.

Additional automatic sharing of metadata added

For administrators: Now when new sites or instruments are added to Madrigal, these metadata files are automatically added to your site.

Experiment level security

Previously, individual Madrigal files could be made public or private. Now entire experiments can be made public, private, or hidden altogether. See the script for details.

Experiment directory naming convention modified

Previous to Madrigal 2.5, all Madrigal experiments had to be stored in directory paths in the form:

$MADROOT/experiments/YYYY/<3 letter inst mnemonic>/ddMMMyy[letter],
Example: /opt/madrigal/experiments/1998/mlh/20jan98b.

Under this convention, the date of the directory name represented the start date of the experiment, with one letter optionally added. This meant there was a limited number of experiments that could be created for a particular day for a particular experiment. This part of the directory naming convention has been dropped, and now the convention is:

$MADROOT/experiments/YYYY/<3 letter inst mnemonic>/*, 
Example: /opt/madrigal/experiments/1998/mlh/mlh_exp_234.

Madrigal 2.4 Release - February 2006

Simple web UI added

A new web user-interface has been added that allows easy printing and plotting of basic Madrigal data. To make it easy to use, advanced Madrigal features such as derived parameters and filtering of data have been removed.

On-demand plot creation

Madrigal now allows users to create basic scatter plots and pcolor plots versus range or altitude of any measured or derived parameter in a data set.

Logging of user data access

Madrigal now logs user's names, emails, and affiliations whenever data files are directly accessed in a file administrators can access.

Automatic updating of all geophysical data

Madrigal now automatically updates all its internal geophysical files (e.g., Kp, Fof2, Dst, Imf, etc) every time updateMaster is run.

Simple-to-use python module to create and edit Madrigal files

There is now a simple-to-use python module to create and edit Madrigal files.

New administrative scripts to manage Madrigal experiments

Administrators can now add or modify all Madrigal experiments using simple administrative scripts, instead of trying to edit Madrigal metadata files themselves or use the complex genExp script.

Complete documentation rewrite

Madrigal documentation has now been completely rewritten and reorganized into three manuals: one for users, one for administrators, and one for developers.

Automatic graphics conversion

Madrigal will now allow users to select any graphics format they prefer for graphics administrators place in experiments. This feature was contributed by Eiscat.

Update of IGRF/MSIS

The Madrigal derivation engine is now using the IGRF 2010 coefficients, and the MSIS 2000 model.

Limiting of disk space used for global search files

Administrators can now limit the maximum amount of disk space used to store temporary global search files. See the section on editing the madrigal.cfg file in the installation guide.



Madrigal 2.3 Release - March 2004

Remote programming access to Madrigal via web services using any platform

Madrigal now exposes all the information and capabilities it has as web services, which allows easy access to Madrigal from any computer on the internet using any platform (Unix, Windows, Mac, etc). Madrigal's web services are basically cgi scripts with simple output that allows easy parsing of the information. Any language that supports the HTTP standard can then access any Madrigal site. We have written remote API's using python and Matlab, but almost any language could be used. See the section on remote programming access for details of these APIs and the underlying web services.

Note that this approach of remotely accessing Madrigal data has been always possible before by parsing the html output meant to be displayed in a web browser (this general programming method is referred to as "screen scraping"). However, not only is this parsing difficult; but the code often breaks when the user interface is modified in any way. With web services the returned cgi scripts are designed to be both simple to parse and stable.

The web services are not implemented according to the SOAP or XMLRPC standard since not all scripting languages have support for these standards (or for XML parsing). Instead they use the simple approach of returning data requested via a query as a delimited text file. These web services are fully documented here.

Users who want only to write programs to remotely access Madrigal, and not to install a Madrigal server themselves, are now able to download the remote python and Matlab API's from the OpenMadrigal site.

Command-line global search

As an example of remote programming access to Madrigal via web services, an application globalIsprint was written using the python remote API that does a global search of data on any Madrigal site that has installed Madrigal version 2.3. This application is installed as part of Madrigal, and also when the standalone remote python API is installed. It has all the filtering ability of the web-based global search.

Calculate any derivable Madrigal parameter for any time and point(s) in space

By clicking on "Run Models", users can calculate any derived Madrigal parameter (such as magnetic fields, or geophysical parameters) for arbitrary times and ranges of position. Note that this capability is also available as a web service, and through the remote python and Matlab API's.

New derived parameters

Bug fixes

The Madrigal C API now no longer aborts when a Cedar file contains cycle marks (Cedar parameter 95) that are not in order. (Reported by Angela Li, SRI)

A problem launching the global search with the python module os.spawnlp was fixed. (Reported by Angela Li, SRI)


Madrigal 2.2 Release - Feb 2003

New derived parameters

Filtering using any parameter

Better help understanding what each parameter means

Improved data output

Better consistency with Cedar standard

New derived parameters are simple to add

Previous: Brief History   Up: Doc home   Next: Madrigal user's guide