Upgrading Madrigal to version 3

The following directions assume you have a version of Madrigal 2 already running, and you want to upgrade it to Madrigal 3. It is recommended you back up your existing Madrigal installation before proceeding. You will also need to upgrade slightly any loading programs you are running to create the new Madrigal 3 Hdf5 CEDAR format. Contact Bill Rideout or go to Create Madrigal files to learn more.

Prerequisites

With Madrigal 3, the installation uses a version of python 3 you install yourself, and you have permission to install into. The easiest way I have found to create and manage python environments with scientific tools is to use the Anaconda python distribution.

Recommended Conda instructions

Follow the Conda installation instructions to install your desired Conda installer (miniforge is recommended)
Initialize/activate Conda
Create your Madrigal Python environment (currently supporting up to Python 3.12) conda create --name madrigal python=3.12

Installation instructions

Be sure the environmental variable MADROOT is set to your main Madrigal directory (where madrigal.cfg is locatied).
Make sure $MADROOT/bin/python links to the version of python listed above. For example, if you are using anaconda, you might run <ln -s //anaconda/bin/python $MADROOT/bin/python>.
Download the latest Madrigal 3 distribution file, madrigal*.tar.gz from the OpenMadrigal distribution page to MADROOT.
tar -xzf madrigal*.tar.gz
Configure your web server to run the Madrigal Django application. Instruction for apache can be found in the Configuring Apache for a Madrigal Django Application section below. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver.
Edit your file $MADROOT/metadata/siteTab.txt. The format of siteTab.txt is described here. Note that if your site will be running https, the third field will begin https://. Make sure the last column of that line for your site ends ,3.0 . This tells all sites that you site is now a Madrigal 3 site. For example:
```
999,mad31,localhost,madrigal,cgi-bin/madrigal,madrigal/servlets,Bill Rideout,MIT Haystack Observatory,Route 40,,Westford,MA,01886,USA,1-781-981-5624,brideout@haystack.mit.edu,3.0
```
Be sure to cd to MADROOT before running the following step. Then you should be able to complete the installation simply by typing
```
bash installMadrigal  &> install.log &
```
Help with any installation errors is available from the OpenMadrigal administrator.
If there were no errors, your madrigal installation should be running at the url given by MADSERVERROOT.
Set the script madroot/bin/updateMaster up as a cron job to run once a day.
If you want to add any documentation pages specific to your site to the Madrigal documentation pages, see the other admin tasks section of this manual.
If you want to add your own rules of the road to the Madrigal experiment page, see the other admin tasks section of this manual.

Editing the madrigal.cfg file

The madrigal.cfg file contains all the configuration information specific to your installation. This section discusses how to edit that file for each parameter. The madrigal.cfg.template file contains examples of each parameter. For an upgrade, this file does not generally need to be edited.

MADROOT - Madrigal root directory (absolute). This must be set as an environmental variable.
MADSERVER - Web server for accessing Madrigal. For https, add https:// to the beginning. If not given, uses http
MADSERVERROOT - document virtual directory relative to MADSERVER. If the full Madrigal Url is the same as the MADSERVER field (for example, if the url is http://madrigal.hao.ucar.edu/), then set this field to a period. (Example: MADSERVERROOT = . )
SITEID - Site ID - Must be unique and same as in siteTab.txt
HTMLSTYLE - Body style of html pages
INDEXHEAD - Heading in the top-level Madrigal page
CONTACT - Mailto link of contact person(s) for this madrigal installation. Multiple email addresses may be included if separated by commas.
MAILSERVER - Name of mailserver (possibly localhost if running sendmail)
PLOTBUTTONLABEL - By default the button to show additional documentation is labelled Plots/Docs. Change this line for a different label.

Configuring Apache for a Madrigal Django Application

Some configuration of apache is required to run a Django application like Madrigal. If you are running a webserver other than apache, please consult the Django documentation for your particular webserver. We will configure apache to run Madrigal in mod_wsgi daemon mode for improved performance.

The first step is to make sure mod_wsgi is installed as part of your apache webserver. The easiest method is to install mod_wsgi in your Madrigal Conda environment conda install mod_wsgi. Once installed, add the line

LoadModule wsgi_module modules/mod_wsgi.so

near the top of the httpd.conf file. Next add the follow section, with MADROOT, MADSERVERROOT and YOUR_SERVER replaced with values for your server from the madrigal.cfg file described below. Note that if MADSERVERROOT is zero length (that is, if the full Madrigal Url is the same as the MADSERVER field - for example, if the url is http://madrigal.hao.ucar.edu/) then /MADSERVERROOT just becomes / unlike in the madrigal.cfg file when it becomes a period. Set NUM_CPU to about one half available on your server.

SetEnv PYTHONPATH MADROOT/source/madpy/djangoMad/
WSGIDaemonProcess YOUR_SERVER python-path=MADROOT/source/madpy/djangoMad processes=NUM_CPU display-name=%{GROUP} lang='en_US.UTF-8' locale='en_US.UTF-8'
WSGIProcessGroup YOUR_SERVER
WSGIScriptAlias /MADSERVERROOT MADROOT/source/madpy/djangoMad/djangoMad/wsgi.py process-group=YOUR_SERVER
Alias /static/ MADROOT/source/madpy/djangoMad/madweb/static/
WSGIApplicationGroup %{GLOBAL}

 <Directory MADROOT/source/madpy/djangoMad/madweb/static>

Require all granted

</Directory>
<Directory MADROOT/source/madpy/djangoMad/djangoMad>

<Files wsgi.py>

Require all granted

</Files>

</Directory>

You may also need to add to your apache configuration file SetEnv LD_LIBRARY_PATH $MADROOT/lib, possibly along with other library paths.

Finally, you need to edit the file MADROOT/source/madpy/djangoMad/djangoMad/settings_production_template.py, and save it as MADROOT/source/madpy/djangoMad/djangoMad/settings_production.py. The lines to be edited are:

SECRET_KEY = '*****' ALLOWED_HOSTS = ['madrigal3.haystack.mit.edu'] ADMINS = (('Bill Rideout', 'brideout@haystack.mit.edu'),) EMAIL_HOST = 'hyperion.haystack.mit.edu' MANAGERS = (('Bill Rideout', 'brideout@haystack.mit.edu'),)

You can generate a new secret key for your site at http://www.miniwebtool.com/django-secret-key-generator/, which is recommended for security reasons. The other lines are self-explanatory.

When you are done, be sure to restart apache for the changes to take effect. Note that because this Madrigal Django application runs in daemon mode, it will be necessary to restart the web server if you ever update the python code.

Note about Apache and mod_wsgi on MacOS

From mod-wsgi PyPi page: "Note that although MacOS X is a UNIX type system, the pip install method is the only supported way for installing mod_wsgi. Since MacOS X Sierra, Apple has completely broken the ability to install third party Apache modules using the apxs tool normally used for this task.

History suggests that Apple will never fix the problem as they have broken things in the past in other ways and workarounds were required as they never fixed those problems either. This time there is no easy workaround as they no longer supply certain tools which are required to perform the installation."

Throttling Madrigal admin emails

Sometimes Madrigal users write scripts that cause numerous admin emails, and in addition malicious users can also generate numerous admin emails. If you would like to throttle the number of Madrigal admin emails you receive, see Throttling admin emails