At SANBI we have a small HPC (see our Annual report) that uses Slurm as a scheduler. Its always a good idea to keep this up to date and unfortunately the version available in the Ubuntu package repository tends to be quite old (e.g. 17.11 for Ubuntu 18.04 and 19.05 for Slurm 20.04).
The Slurm upgrade procedure is mentioned in their Quick Start Administrator Guide. In short, the daemons need to be upgraded in a specific order, starting with slurmdbd, followed by a nested upgrade of slurmctld and the slurmd on each compute node. To facilitate this process our install is on shared storage (CephFS, but could also be NFS) and looks as follows:
/tools/admin/slurm ├── 18.08.9 ├── 19.05.7 ├── 20.02.5 ├── ctld -> /tools/admin/slurm/20.02.5 ├── current -> /tools/admin/slurm/20.02.5 ├── d -> /tools/admin/slurm/20.02.5 ├── dbd -> /tools/admin/slurm/20.02.5 ├── etc
To install slurm, the slurm source is unpacked and compiled with slurm, with the configure options like:
./configure --prefix=/tools/admin/slurm/20.02.5 --sysconfdir=/tools/admin/slurm/etc
As can be seen from the above listing, the d
, ctld
, dbd
and current
links link to the current version of Slurm in use.
Each daemon is managed by systemd and configured with a file in /etc/systemd/system
. For example here is the configuration of slurmctld
(i.e. /etc/systemd/system/slurmctld
):
[Unit] Description=Slurm controller daemon After=network.target munge.service ConditionPathExists=/tools/admin/slurm/etc/slurm.conf [Service] Type=oneshot EnvironmentFile=-/etc/sysconfig/slurmctld ExecStartPre=-/usr/bin/pkill -KILL slurmctld ExecStart=/tools/admin/slurm/ctld/sbin/slurmctld $SLURMCTLD_OPTIONS ExecReload=-/bin/kill -HUP $MAINPID ExecStop=-/usr/bin/pkill -KILL slurmctld PIDFile=/var/run/slurm/slurmctld.pid KillMode=process LimitNOFILE=51200 LimitMEMLOCK=infinity LimitSTACK=infinity RemainAfterExit=true [Install] WantedBy=multi-user.target
After installing this file, you need to run sudo systemctl daemon-reload
.
Note the line in bold in the above config file. The executable is run via the /tools/admin/slurm/ctld/sbin
folder. Because this /tools/admin/slurm/ctld
path is a symlink, upgrading the slurmctld
involves simply changing the symlink to point to the new slurm version.
The upgrade process for slurmdbd
and slurmctld
is quite straightforward, just follow the procedure for database backup and upgrade as mentioned in the docs. For slurmd
, the upgrade procedure (backup of StateSaveLocation
and restart of slurmd
) needs to happen on each worker node. This can best be automated using ansible
. As noted in the Slurm admin documentation, you can at most upgrade between two major releases. Due to a security issue older Slurm versions are not available from the main download page but you can still get them from Github (e.g. version 19.05).
A final note - this procedure, with per-version symlinks etc, was based on something I read online before executing at SANBI. I can't recall where I read this but if you were the source and would like credit, please look me up and let me know.
P.S. after the system upgrade, I recompile and re-install the slurm-drmaa module that you can find here.