We are preparing for Galaxy Africa in a few weeks’ time, which will feature some Galaxy training. In preparation for that I installed a new Ubuntu 16.04 virtual machine to host a 18.01 Galaxy server. The aim is to set up a production Galaxy server. To that end, the server is being hosted on a 1 TB Ceph RBD partition mounted on
/galaxy. A user called
galaxyuser was created on our FreeIPA authentication environment, and
/galaxy/galaxysrv was created to host Galaxy files.
The first step of setup was to clone Galaxy 18.01 release and configure it for production use. The postgresql database server was installed and a user created for
galaxyuser and then that user used to create the
galaxy database. I configured the database and added myself (
email@example.com) as an admin user.
The next step was to install
nginx. As far as possible I tried to not alter the “out of the box” nginx configuration, to make it easier to do upgrades later. To that end, firstly, a SSL certificate was added using
certbot and Let’s Encrypt by installing the
python-certbot-nginx packages, and running
certbox --nginx certonly. This yielded
/etc/letsencrypt/live/galaxy.sanbi.ac.za and associated files. The
/etc/nginx/ssl directory as created and a
/etc/nginx/ssl/dhparam.pem file was created with
openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096. This was in order to create a more secure configuration than default as explained here.
Following the instructions from the Galaxy Receiving Files with nginx documentation and advice from Marius van den Beek,
nginx-extras was installed from the recommended PPA, yielding
nginx-extras packages for version
1.10.3-0ubuntu0.16.04.2ppa1. Then a file
/etc/nginx/conf.d/custom.conf was created with content as per this gist. This is effectively a combination of the options suggested by the Galaxy admin docs with those in
/etc/letsencrypt/options-ssl-nginx.conf. The server configuration directives from the recommended Galaxy configuration were adapted and put in
/etc/nginx/sites-available/galaxy. The resulting configuration is in this gist. Once added, the configuration was activated by removing the
/etc/nginx/sites-enabled/default file and linking the
galaxy configuration fie in its place. Finally,
/etc/nginx/nginx.conf was altered by changing the user used to run the server to
galaxyuser (i.e. “user galaxyuser”). To connect Galaxy to nginx, the
socket: option in the Galaxy
config/galaxy.yml and the configuration in the nginx site configuration were harmonised as per the relevant documentation. Since the unix socket was not created on startup, a http connection and thus TCP socket on localhost was used.
The third step was configuring Galaxy to start using
supervisord. This was based on the [program:web] configuration from the Galaxy starting and stopping configuration guide. And this is where things started going wrong. Using this configuration, the data upload tool didn’t work as it used the system Python, not Python from the Galaxy virtualenv configured in
/galaxy/galaxysrv/galaxy/.venv. To ensure that the Galaxy
virtualenv was activated before running the upload tool, the VIRTUAL_ENV config was added to the
/etc/supervisor/conf.d/galaxy configuration, resulting in the config shown in this gist.
The fourth step was to configure CVMFS to allow access to the reference data collection used on usegalaxy.org. I installed the
cvmfs package by following the instructions to install the apt repository and then
apt-get install cvmfs. The correct configuration was learned from Björn Grüning (@bgruening)’s
bgruening/galaxy-stable Docker container with some help from @scholtalbers on gitter:
/etc/cvmfs/domain.d/galaxyproject.org.conf put the line as per this gist.
CVMFS_REPOSITORIES="data.galaxyproject.org" CVMFS_HTTP_PROXY="DIRECT" CVMFS_QUOTA_LIMIT="4000" CVMFS_USE_GEOAPI="yes"
-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5LHQuKWzcX5iBbCGsXGt 6CRi9+a9cKZG4UlX/lJukEJ+3dSxVDWJs88PSdLk+E25494oU56hB8YeVq+W8AQE 3LWx2K2ruRjEAI2o8sRgs/IbafjZ7cBuERzqj3Tn5qUIBFoKUMWMSIiWTQe2Sfnj GzfDoswr5TTk7aH/FIXUjLnLGGCOzPtUC244IhHARzu86bWYxQJUw0/kZl5wVGcH maSgr39h1xPst0Vx1keJ95AH0wqxPbCcyBGtF1L6HQlLidmoIDqcCQpLsGJJEoOs NVNhhcb66OJHah5ppI1N3cZehdaKyr1XcF9eedwLFTvuiwTn6qMmttT/tHX7rcxT owIDAQAB -----END PUBLIC KEY-----
d. Add the line
/cvmfs /etc/auto.cvmfs to
/etc/auto.master yielding a file looking like this gist.
/cvmfs directory was created to be a mount point (
f. The autofs service was restarted (
systemctl restart autofs) and then a
ls /cvmfs/data.galaxyproject.org/byhand shows a pretty collection of reference data.
g. updated The
config/galaxy.yml file was updated so that the tool_data_table_config_path key contains references to the files that are stored in CVMFS. The final value of this key was:
This might not fit on your screen, so see the config fragment here. After the update to the Galaxy config all service were restarted (with
sudo supervisorctl restart all).
My fifth and final step was to test the Galaxy server by installing bowtie2 from the toolshed and working through the first steps of the mapping tutorial. Both human (hg19) and fruitfly (dm3) reference genomes were downloaded (and apparently stored in
/var/lib/cvmfs/shared) using CVMFS and the bowtie2 mapping was run against them successfully, yielding the results expected from the tutorial.
Future work? There is lots – I have to connect the server to a cluster, using Slurm, and enable Interactive Environments… I’ll blog about that when I get there.