I recently switched out local Galaxy server to be run using uWSGI and supervisord instead of the standard
run.sh (which uses Paste under the hood). I followed the Galaxy scaling guide and it was pretty accurate except for a few details. I won’t be showing the changes to Galaxy config files, they are exactly as related on that page.
supervisord by doing
pip install supervisor in the virtualenv that Galaxy uses. Then I put a
supervisord.conf in the
config/ directory of our Galaxy install and it starts like this:
[inet_http_server] port=127.0.0.1:9001 [supervisord] [supervisorctl]
[inet_http_server] section directs
supervisord to listen on localhost port 9001. The following two sections,
[supervisorctl] need to be present but can be empty. The rest of the configuration is as per that on the Scaling page with a few changes I’ll explain below:
[program:galaxy_uwsgi] command = /opt/galaxy/.venv/bin/uwsgi --plugin python --ini-paste /opt/galaxy/config/galaxy.ini --die-on-term directory = /opt/galaxy umask = 022 autostart = true autorestart = true startsecs = 10 user = galaxy environment = PATH=/opt/galaxy/.venv:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,PYTHON_EGG_CACHE=/opt/galaxy/.python-eggs,PYTHONPATH=/opt/galaxy/eggs/PasteDeploy-1.5.0- py2.7.egg,SGE_ROOT=/var/lib/gridengine numprocs = 1 stopsignal = TERM [program:handler] command = /opt/galaxy/.venv/bin/python ./scripts/paster.py serve config/galaxy.ini --server-name=handler%(process_num)s --pid-file=/opt/galaxy/handler%(process_num)s.pid --log- file=/opt/galaxy/handler%(process_num)s.log directory = /opt/galaxy process_name = handler%(process_num)s numprocs = 2 umask = 022 autostart = true autorestart = true startsecs = 15 user = galaxy environment = PYTHON_EGG_CACHE=/opt/galaxy/.python-eggs,SGE_ROOT=/var/lib/gridengine
SGE_ROOT is necessary because our cluster uses Sun Grid Engine and the SGE DRMAA library requires this environment variable. Otherwise this config uses uWSGI installed (using
pip) in the virtualenv that Galaxy uses.
This snipped of
nginx configuration shows what was commented out and what was added to link
nginx to uWSGI:
#proxy_set_header REMOTE_USER $remote_user; #proxy_set_header X-Forwarded-Host $host; #proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; #proxy_set_header X-URL-SCHEME https; #proxy_pass http://galaxy_app; uwsgi_pass 127.0.0.1:4001; uwsgi_param UWSGI_SCHEME $scheme; include uwsgi_params;
Then, how to start and stop it all? Firstly, the
supervisord config. The basis for this was the
debian-norrgard script from the supervisord initscripts repository. The final script is in this gist. Note these lines:
supervisord to Galaxy settings. Then
/etc/init.d/galaxy is in this gist. It depends on the
supervisord startup script and starts and stops Galaxy using
Two things remain unsatisfactory:
The shutdown of Galaxy doesn’t work reliably. The use of uWSGI’s
stopsignal = TERMin the
supervisord.confis an attempt to remedy this.
The uWSGI config relies on the
PasteDeployegg. This exists on our Galaxy server because it was downloaded by the historical Galaxy startup script. With the switch towards wheel based (instead of egg based) packages, this script is no longer part of a Galaxy install. The uWSGI settings might need to be changed because of this, however, the
PasteDeploypackage is installed in the virtualenv that Galaxy uses, so perhaps no change is necessary. I haven’t tested this.
With these limitations, however, our Galaxy server is working and much more responsive than before.