Deploying the Renku platform

This guide will help you deploying and customizing your own Renku instance in a cluster.

Cluster requirements

To deploy Renku in a cluster, you need to have the following prerequisites:

Optionally, you can choose to manage your own instances of the following:

Officially supported versions

The following component versions have been tested in production.

Renku version

Kubernetes

Helm

0.6.x

1.14, 1.15

2

0.7.x

1.16, 1.17

3

0.8.x

1.19, 1.20

3

Pre-deployment steps

1. (Optional) Stand-alone configurations

2. Create and configure Renku PVs and PVCs

All of Renku information is stored in three volumes needed by Jena, Postgresql and GitLab (if not stand-alone). To this end, persistent volumes and persistent volume claims need to be created. You can leave k8s to dynamically provision these volumes but we advise to create them beforehand and make regular backups. You can use the following manifest files as examples and apply them through kubectl.

$ kubectl create -f renku-pv.yaml
$ kubectl create ns renku
$ kubectl -n renku create -f renku-pvc.yaml

Ensure that PVs are added correctly and have the appropriate storage class.

3. Create a renku-values.yaml file

This step is very important as it will define the values that helm will use to install your Renku instance.

Basic configuration

The basic Helm configuration file is in the Renku repository: Renku values file

You can use the make-values.sh script to generate automatically a minimal Renku values file. You need to provide your DNS and (optionally) your Gitlab URL, application ID and secret. This script needs bash >=4.0 and openssl installed and the minimal-values.tmpl file

For MacOS: .. code-block:: bash

$ brew install bash $ brew install openssl

$ ./make-values.sh -d renku.mydomain.ch -o renku-values.yaml

After preparing your values and before deploying Renku please make sure to check the following in the values file:

  • the URLs and DNS names are correct

  • all the necessary secrets are configured

  • the ingress configuration is correct

  • add/verify, if necessary, the persistent volume claims

Further configuration

You might want to modify some default values, here are some examples:

  • Logged-out users can be given the permission to launch interactive sessions from public notebooks. This feature is turned off by default. For details on how to enable this feature, see the dedicated section below.

  • A custom privacy notice and cookie banner can be configured (as of Renku 0.6.8). See the dedicated section below.

  • Resources requests and limits can be adjusted for the diverse RenkuLab components

  • For deployments that include GitLab you can configure an S3 backend for lfs objects and registry, this configuration can be added in the gitlab section

  • It is possible to dedicate some nodes for running interactive user sessions.

Deploying RenkuLab

1. (Optional) Certificates

If you chose to create a certificate manually instead of using LetsEncrypt or similar, you can create the corresponding TLS secret with the following command:

$ kubectl -n renku create secret tls renku-mydomain-ch-tls --cert=certificate.crt --key=certificate.key

Note that renku-mydomain-ch-tls should correspond to the ingress TLS value in Renku values file

2. Deploy RenkuLab

Once all the pieces are in place, you can deploy Renku with the following commands:

$ helm init
$ helm repo add renku https://swissdatasciencecenter.github.io/helm-charts/
$ helm upgrade --install renku renku/renku \
 --namespace renku \
 -f renku-values.yaml \
 --timeout 1800s

During deployment you can check the Renku pods being started. If deploying for the first time, GitLab and Keycloak take around 15 minutes to get started. It is normal to see other pods of components that depend on these to be in a CrashLoopBackOff state.

Once all the pods are correctly running or completed, a GitLab Runner can be configured.

3. Verifying RenkuLab deployment

Check list:

  • Helm command completed successfully.

  • RenkuLab pods should be all running and jobs completed.

  • RenkuLab pods logs should not show errors.

  • Perform a quick check:

    1. go to your RenkuLab instance domain (this will verify the ingress is working properly)

    2. login with a valid user, you can register a new one (this will verify various authentication steps)

    3. create a project (this will verify further authentication, templates and GitLab runner)

    4. add a dataset (this will verify the core backend, LFS objects, and Knowledge Graph components work)

    5. launch an environment (this will verify the notebooks component and GitLab image registry)

  • You should be now able to follow “First steps”

If some Renku pods are not starting or present some errors please check the troubleshooting page.

4. Acceptance tests (optional)

If the Helm deployment ran successfully according to the check list above, you might want to run Renku’s acceptance tests. You simply need to:

  • create a user for testing (with no admin rights) and optionally have an S3 bucket credentials for saving screenshots in case a test fails.

  • fill in the credentials (notably email and password) in the test section in the values file and update the deployment

$ helm upgrade --install renku renku/renku \
 --namespace renku \
 --version <renku-version> \
 -f renku-values.yaml \
 --timeout 1800s

  • run the tests. This will create a test pod that needs around 4GB of RAM available. Note that running the complete acceptance test suit takes around 40 minutes to complete.

$ helm test renku \
 --namespace renku \
 --log-file renku-tests.log \
 --timeout 80m

5. Post deployment configurations

After RenkuLab has been deployed you can make some post deployment configurations.

GitLab deployed as part of Renku

If your RenkuLab deployment includes GitLab you need to follow some additional steps to configure an admin user on GitLab.

  1. turn off automatic redirection to GitLab by redeploying with the value gitlab.oauth.autoSignIn: false

  2. log in as the root user using the password from gitlab.password

  3. modify any users you want to modify (e.g. to make them admin)

  4. turn the autoSignIn setting back on

Independent GitLab as identity provider

If the RenkuLab deployment is relying on an existing GitLab instance, this instance could be configured as an identity provider for RenkuLab’s keycloak. This way all of the users from the existing GitLab instance can log into RenkuLab with their existing login information.

  1. Go to https://<renku-domain>/auth (or to your stand-alone Keycloak dashboard), login to Admin console using admin as username and the decoded password stored in keycloak-password-secret.

  2. Add an Identity Provider of type OpenID Connect v1.0.

  3. Set Alias to <renku-domain>, Authorization URL to https://<gitlab-domain>/oauth/authorize, Token URL to https://<gitlab-domain>/oauth/token and Client ID and Client Secret to the values used in step 1.

  4. Click Save. The new Identity Provider should appear and any user from the stand-alone GitLab instance should be able to login.

Upgrading RenkuLab

To upgrade RenkuLab to a newer version please check the Upgrading from 0.x.x section of the release notes in case the renku-values.yaml file needs to be modified. Following, you can upgrade Renku via Helm.

$ helm repo update
$ helm upgrade --install renku renku/renku \
 --namespace renku \
 --version <renku-version> \
 -f renku-values.yaml \
 --timeout 1800s