Running Renku on HPC
The Renku CLI supports various backends for executing workflows. Currently, two
different providers are implemented, namely
toil. If you
have a specific provider you need for your infrastructure, have a look at
Implementing a workflow provider for a detailed description of how to implement
your own workflow provider. Alternatively, please make a feature request.
By default all workflows are executed by the
cwltool provider, that
exports the workflow to CWL and then uses cwltool
to execute the given CWL.
The workflow backend can be changed by using the
command line option. A backend’s default configuration can be overridden by
-c/--config <config.yaml> command line parameter.
renku commands support the above mentioned workflow provider
related command line options:
For example, to execute a previously created
my_plan workflow with
would simply run the following command:
$ renku workflow execute -p toil my_plan
In order to use any other provider with
renku you must first install the required
extras. In the case of
toil this means running the install command like
$ pip install renku[toil]
Renku on Slurm
Slurm is a highly configurable open-source workload manager, which is in widespread use at government laboratories, universities and companies world wide. It is used in over half of the top 10 systems in the TOP500 listing.
toil supports Slurm, one can easily execute the previously created renku
workflows on a Slurm-managed HPC resource. One just needs to provide a simple
configuration file to the provider (
disableCaching option is necessary for Slurm; for more details see the
toil issue TOIL-1006.
Additional Slurm-specific parameters can be
provided with the
TOIL_SLURM_ARGS environment variable.
Taking the example above, the following command line will execute
my_plan on Slurm:
$ TOIL_SLURM_ARGS="-A my_account --export=ALL" renku workflow execute -p toil -c provider.yaml my_plan
provider.yamlfile contains the above mentioned two parameters for
-A my_accountspecifies which account should be charged for the used resources,
--export=ALLspecifies that all environment variables are propagated to the Slurm workers. It is often required in academic Slurm installations.
Run with Docker Containers
You can execute workflows using Docker with
toil as it supports Docker
containers. To do that, you need to add a
docker section in the provider’s
config file and define a Docker image to be used for execution:
toil plugin then uses the specified image to run your workflow. For
a list of available configuration options see Docker API
documentation. Refer to Toil’s Docker documentation
for more information on how to set up Docker for use in
Note that due to the limitations in the underlying libraries, workflows that
use redirection from/to
stdin cannot be executed with Docker.
stdout is supported.
By default, if the config file doesn’t define a
Renku sets a default working directory and creates a volume in the container
and mounts project’s directory to it to make sure that the container can access
local files. However, if you define a
working_dir in the config file,
Renku doesn’t create this volume automatically and you must make sure that
local files required to execute a workflow are accessible in the container.