Installing Madrigal for the first time

Prerequisites

Madrigal is meant to be installed on any Unix-like OS, including Mac and Linux. Most of the prerequisites for Madrigal are pre-installed in any modern unix distribution. They are:

A C and a FORTRAN compiler. The free GNU compilers may be downloaded from the GNU Website .
Autotools (automake, autoconf, aclocal, libtoolize, autoheader), along with h5repack (may be part of h5utils package).
A web server. See instructions below to run a Django application (which Madrigal is) on apache.
The environmental variable MADROOT should be set to the path to the Madrigal installation directory. Its recommended this be done in your login script (eg, .bash_profile) because you will need it to run administrative scripts later.
With Madrigal 3, the installation uses a version of python 3 you install yourself, and you have permission to install into. The easiest way I have found to create and manage python environments with scientific tools is to use the Anaconda python distribution.

Recommended Conda instructions

Follow the Conda installation instructions to install your desired Conda installer (miniforge is recommended)
Initialize/activate Conda
Create your Madrigal Python environment (currently supporting up to Python 3.12) conda create --name madrigal python=3.12

Installation instructions

Madrigal can be installed on any unix server with a web server configured for Django. If you want to link your data in with data on other Madrigal servers, please notify the OpenMadrigal administrator. In general you do not need root permission to install madrigal once the prerequisites listed above are installed.

Create a directory to be used as your Madrigal root directory on your unix server. This directory will be referred to as MADROOT.
Create the environment variable MADROOT with that directory path.
Download the latest Madrigal distribution file, madrigal*.tar.gz from the OpenMadrigal distribution page to MADROOT.
gunzip madrigal*.tar.gz
tar -xf madrigal*.tar
Repeat the 3 steps above to get and untar the Sample Experiments file experiments.tar.gz.
Make sure $MADROOT/bin/python links to the version of python listed above. For example, if you are using anaconda, you might run <ln -s //anaconda/bin/python $MADROOT/bin/python>.
If desired, copy any data from your old $MADROOT/experiments directory to this new $MADROOT/experiments directory.
In madroot, there will now be a file called madrigal.cfg.template. Copy it to madrigal.cfg, and edit all the configuration parameters, as described in the Editing madrigal.cfg section below.
Configure your web server to run the Madrigal Django application. Instruction for apache can be found in the Configuring Apache for a Madrigal Django Application section below. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver.
Edit the file MADROOT/metadata/siteTab.txt with a unique id to include your site. The format of siteTab.txt is described here. Note that if your site will be running https, the third field will begin https://. If you want to be an official Madrigal site that other Madrigal sites share data with, request the OpenMadrigal administrator to assign you this id. If this is just a test site, use any id not already in the siteTab.txt file. Finally, make sure the last column of the line for your site ends ,3.0 . This tells all sites that you site is now a Madrigal 3 site. For example:
```
999,mad31,localhost,madrigal,cgi-bin/madrigal,madrigal/servlets,Bill Rideout,MIT Haystack Observatory,Route 40,,Westford,MA,01886,USA,1-781-981-5624,brideout@haystack.mit.edu,3.0
```
Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing
```
bash installMadrigal  &> install.log &
```
There may be a long pause when running updateMaster near the end of the installation since the instParmTab.txt metadata file is being built for the first time by examining every data file, but future calls to updateMaster will be much faster since only new experiments are examined. Help with any installation errors is available from the OpenMadrigal administrator.
If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT.
Set the script madroot/bin/updateMaster up as a cron job to run once a day.
If you want to add any documentation pages specific to your site to the Madrigal documentation pages, see the other admin tasks section of this manual.
If you want to add your own rules of the road to the Madrigal experiment page, see the other admin tasks section of this manual.

Editing the madrigal.cfg file

The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file for each parameter. The madrigal.cfg.template file contains examples of each parameter.

MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable.
MADSERVER - Web server for accessing Madrigal. For https, add https:// to the beginning. If not given, uses http
MADSERVERROOT - document virtual directory relative to MADSERVER. If the full Madrigal Url is the same as the MADSERVER field (for example, if the url is http://madrigal.hao.ucar.edu/), then set this field to a period. (Example: MADSERVERROOT = . )
SITEID - Site ID - Must be unique and same as in siteTab.txt
HTMLSTYLE - Body style of html pages
INDEXHEAD - Heading in the top-level Madrigal page
CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas.
MAILSERVER - Name of mailserver (possibly localhost if running sendmail)
PLOTBUTTONLABEL - By default the button to show additional documentation is labelled Plots/Docs. Change this line for a different label.

Configuring Apache for a Madrigal Django Application

Some configuration of apache is required to run a Django application like Madrigal. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. We will configure apache to run Madrigal in mod_wsgi daemon mode for improved performance.

The first step is to make sure mod_wsgi is installed as part of your apache webserver. The easiest method is to install mod_wsgi in your Madrigal Conda environment conda install mod_wsgi. Once installed, add the line

LoadModule wsgi_module modules/mod_wsgi.so

near the top of the httpd.conf file. Next add the follow section, with MADROOT, MADSERVERROOT and YOUR_SERVER replaced with values for your server from the madrigal.cfg file described below. Note that if MADSERVERROOT is zero length (that is, if the full Madrigal Url is the same as the MADSERVER field - for example, if the url is http://madrigal.hao.ucar.edu/) then /MADSERVERROOT just becomes / unlike in the madrigal.cfg file when it becomes a period. Set NUM_CPU to about one half available on your server.

SetEnv PYTHONPATH MADROOT/source/madpy/djangoMad/
WSGIDaemonProcess YOUR_SERVER python-path=MADROOT/source/madpy/djangoMad processes=NUM_CPU display-name=%{GROUP} lang='en_US.UTF-8' locale='en_US.UTF-8'
WSGIProcessGroup YOUR_SERVER
WSGIScriptAlias /MADSERVERROOT MADROOT/source/madpy/djangoMad/djangoMad/wsgi.py process-group=YOUR_SERVER
Alias /MADSERVERROOT/static/ MADROOT/source/madpy/djangoMad/madweb/static/
WSGIApplicationGroup %{GLOBAL}

 <Directory MADROOT/source/madpy/djangoMad/madweb/static>

Require all granted

</Directory>
<Directory MADROOT/source/madpy/djangoMad/djangoMad>

<Files wsgi.py>

Require all granted

</Files>

</Directory>

You may also need to add to your apache configuration file SetEnv LD_LIBRARY_PATH $MADROOT/lib, possibly along with other library paths.

Finally, you need to edit the file MADROOT/source/madpy/djangoMad/djangoMad/settings_production_template.py, and save it as MADROOT/source/madpy/djangoMad/djangoMad/settings_production.py. Note that if MADSERVERROOT is zero length then /MADSERVERROOT/static just becomes /static. The lines to be edited are:

SECRET_KEY = '*****' ALLOWED_HOSTS = ['madrigal3.haystack.mit.edu'] ADMINS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) EMAIL_HOST = 'hyperion.haystack.mit.edu' MANAGERS = (('Bill Rideout', 'brideout@haystack.mit.edu'),)

You can generate a new secret key for your site at http://www.miniwebtool.com/django-secret-key-generator/, which is recommended for security reasons. The other lines are self-explanatory.

When you are done, be sure to restart apache for the changes to take effect. Note that because this Madrigal Django application runs in daemon mode, it will be necessary to restart the web server if you ever update the python code.

Note about Apache and mod_wsgi on MacOS

From mod-wsgi PyPi page: "Note that although MacOS X is a UNIX type system, the pip install method is the only supported way for installing mod_wsgi. Since MacOS X Sierra, Apple has completely broken the ability to install third party Apache modules using the apxs tool normally used for this task.

History suggests that Apple will never fix the problem as they have broken things in the past in other ways and workarounds were required as they never fixed those problems either. This time there is no easy workaround as they no longer supply certain tools which are required to perform the installation."

Throttling Madrigal admin emails

Sometimes Madrigal users write scripts that cause numerous admin emails, and in addition malicious users can also generate numerous admin emails. If you would like to throttle the number of Madrigal admin emails you receive, see Throttling admin emails