I recently upgraded one of my Galaxy servers to version 22.05 (once again using Ansible and the excellent Galaxy Ansible roles). Unfortunatey this exposed a problem with our environment. The Galaxy server that I configured is running on Ubuntu 20.04, but the HPC cluster's worker nodes use Ubuntu 18.04. Ubuntu 18.04 comes with Python 3.6, which is too old for modern Galaxy.
Thanks to advice from Galaxy admin's chat I installed a Conda environment by downloading Miniconda3 and doing a batch install (
Miniconda3-latest-Linux-x86_64.sh -b -p GALAXY_DIR/conda, where GALAXY_DIR is my Galaxy installation directory). Then I activated this with
source GALAXY_DIR/conda/bin/activate) and proceeded to run Galaxy's
common_startup.sh as described in this post. My GALAXY_VIRTUAL_ENV in
job_conf.xml points to the location of the virtualenv.
The end result is that Galaxy uses this virtual environment to provide Python on cluster nodes, and job submission now works again.