Renku Architecture

This document describes the architecture of the services that are parts of the Renku platform.

The Renku platform consists of several off-the-shelf components from the software engineering and data science software stacks, as well as customized or newly developed services. The services communicate among each other through REST APIs. The deployment is orchestrated on Kubernetes through the use of Helm charts. Because of its micro-service architecture, Renku is by definition modular, and therefore able to accommodate other languages or implementations used for individual components.

Components

Renku has the following custom components:

  • renku: the meta repository with deployment scripts, documentation and kubernetes helm charts
  • renku-gateway: an API gateway connecting clients to the APIs of the different backend services
  • renku-graph: a collection of services concerned with activating, building and querying the Renku knowledge graph
  • renku-notebooks: a service integrating GitLab repositories with JupyterHub
  • renku-python: python API & Command Line Interface (CLI)
  • renku-ui: web front-end interface

In addition, we make use of:

  • JupyterHub: management of interactive notebook servers
  • GitLab: repository management, CI and various related APIs
  • Keycloak: identity management and user authentication

The figure below shows an overview of the components and their interactions. Blue components are off-the-shelf, yellow components are either built or extended by us.

strict digraph architecture { compound=true; newrank=true; // ordering=out; graph [fontname="Raleway", nodesep="0.8"]; node [shape="rect", style="filled,rounded", fontname="Raleway"]; edge [fontname="Raleway"] # main off-the-shelf services GitLab [fillcolor="lightblue" URL="../developer/services/notebooks_service.html#image-builds" target="_graphviz"] Keycloak [fillcolor="lightblue"] JupyterHub [label="Notebook service" fillcolor="#f4d142" URL="../developer/services/notebooks_service.html" target="_graphviz"] Notebook [fillcolor="lightblue"] # clients UI [fillcolor="#f4d142"] CLI [fillcolor="#f4d142"] gateway [fillcolor="#f4d142" URL="../developer/services/api_gateway.html" target="_graphviz"] "Knowledge Graph" [fillcolor="#f4d142" URL="../developer/services/graph_services.html" target="_graphviz"] storage [fillcolor="lightblue", label="Object store", shape="cylinder"] subgraph cluster_clients { label="Clients" UI CLI Notebook {rank=same; UI, CLI, Notebook}; } JupyterHub -> Notebook [label=" spawn"] CLI -> GitLab [label=" git pull/push"] Notebook -> GitLab [label=" git pull/push"] Notebook -> storage [label=" pull/push LFS\n fetch image"] UI -> gateway [label=" API"] GitLab -> Keycloak [label=" OAuth"] JupyterHub -> GitLab [label=" OAuth"] gateway -> GitLab [label=" proxy API"] gateway -> JupyterHub [label=" proxy API"] gateway -> Keycloak [label=" OIDC"] GitLab -> storage [label=" fetch/push image"] CLI -> storage [label=" pull/push LFS"] GitLab -> "Knowledge Graph" [label=" webhook"] gateway -> "Knowledge Graph" [label=" GraphQL"] {rank=0; UI} }

Important Concepts

In order to develop for the platform, it is strongly advised to get familiarized with a few foundational concepts: