Brief history of Madrigal

Madrigal is a database of ground-based measurements and models of the Earth's upper atmosphere and ionosphere. It is the community database of the Coupling, Energetics and Dynamics of Atmospheric Regions (CEDAR) program, which is devoted to the characterization and understanding of the atmosphere above about 60 km, with emphasis on the various processes that determine the basic structure and composition of the atmosphere, and on the mechanisms that couple different atmospheric regions. Instruments developed or upgraded under CEDAR include interferometers, spectrometers, imagers, lidars and medium, high-frequency and incoherent scatter radars. The success of CEDAR has been due, in large measure, to its ability to encourage collaborative efforts coalescing observations, theory and modeling. The CEDAR community includes about 800 scientists and students from around the world.

From the inception of the CEDAR program in 1988, there has been a great concern among the members of the CEDAR community to make the data collected by the CEDAR instruments easily accessible for joint studies. Consequently, a high priority was placed on establishing a repository for CEDAR data and model results. An incoherent scatter radar database had been established at the National Center for Atmospheric Research (NCAR) in 1985, and this evolved into the CEDAR Database in 1989. By the end of 1997, it had grown to include data from 44 instruments and 16 models. Over 200 users have requested information from the Database. In 2012, the Madrigal database and the CEDAR database were combined into a single database system, the Madrigal CEDAR database, based at MIT Haystack Observatory.

A central element of the CEDAR Database is a standard data format. With Madrigal 3.0 the file format is now Hdf5, with a well-defined layout and parameters. This new CEDAR Hdf5 format is described here. This format replaced the old 16 bit integer based format that evolved from the format used by the earlier incoherent scatter database, which in turn evolved from an earlier version of Madrigal developed at the MIT Haystack Observatory in 1980.

Haystack maintains and develops the Madrigal CEDAR database as an open-source project with community contributions. With the Madrigal CEDAR database, the site owner stores only their own data, which they can add to or update at any time. However, because the Madrigal database shares its metadata with all other Madrigal sites, users browsing any Madrigal site can search for data at any other Madrigal site. In addition, a central archival Madrigal site (cedar.openmadrigal.org) archives all Madrigal sites and stores data not stored at other Madrigal sites.

Madrigal data are arranged into "experiments", which may contain data files, images, documentation, links, etc. A key feature of Madrigal is its seamless integration of archival and real-time data. A realtime file on Madrigal is accessed in exactly the same way as any archival file.

Madrigal has been installed at numerous locations in addition to Millstone Hill, including EISCAT, SRI International, and Jicamarca. The inventories of experiments available at each installation are available to the other installations through shared metadata. New Madrigal sites can be automatically added at any time.

Madrigal is an open source project with a central developer's web site and a central repository. A complete Subversion archive of all Madrigal software, including software which is not included with the standard distribution, is available at the developer's web site. There is also an Open Madrigal mailing list and developer's forum. Any group wishing to install Madrigal to distribute their instrument's data is welcome to - see the http://cedar.openmadrigal.org/openmadrigal web site for details. The central repository link is a place where all data from all Madrigal sites is archived. It can also be used to access data, although accessing the local sites is preferred.