Core service

A lot of Renku core functionality is sometimes needed to be accessed via HTTP or executed in the background. The main purpose of core service is to address those needs.

The following diagram describes components and architecture of the service:

strict digraph architecture { compound=true; newrank=true; rankdir=LR; graph [fontname="Raleway", nodesep="0.8"]; node [shape="rect", style="filled,rounded", fontname="Raleway"]; edge [fontname="Raleway"] Service [fillcolor="lightblue"] Worker [fillcolor="lightblue"] "Filesystem Cache" [fillcolor="#f4d142"] "Memory Cache" [fillcolor="#f4d142"] GitLab [fillcolor=""] Worker -> "Filesystem Cache" [label=" execute operation on cached resource"] Worker -> "Memory Cache" [label=" read/write resource metadata"] subgraph service_clients { label="Clients" UI -> Service SDK -> Service {rank=same; UI, SDK}; } Service -> "Filesystem Cache" [label=" persist resource"] Service -> "Memory Cache" [label=" persist resource metadata"] Service -> GitLab [label=" sync with remote"] Worker -> GitLab [label=" sync with remote"] subgraph cluster_service_cache { color="lightgray"; label="Service Cache"; "Filesystem Cache" [fillcolor="#f4d142"] "Memory Cache" [fillcolor="#f4d142"] } subgraph cluster_service_client { color="lightgray"; label="Clients"; UI SDK } }

The resources are segmented by user and can not be shared between them. They could be files or projects on top of which the user executes subsequent operations. Each resource within the service has a default time to live after which the resource is evicted.

Sequence diagram of service APIs

POST /<prefix>/cache.project_clone

An endpoint to clone remote project to service cache. Used as a first step in the flow for executing operations on top of the cloned project.

../../_images/9e638547ef315202eb923f0e802344619baf076186c290b701407cb98c15cd14.png

GET /<prefix>/cache.project_list

An endpoint to list all projects to the service cache for a given user.

../../_images/5b3227daa075385ce708f4f402b88ee3b126edaee54a7c2eed789cb706691f5e.png

POST /<prefix>/cache.files_upload

An endpoint that allows file upload and tracks it for each user using the service.

../../_images/39a684df06d38e25ae5ec9da5d193e4f45f4c21daaeac6fb2f27a519a0cdc88d.png

GET /<prefix>/cache.files_list

An endpoint to list all uploaded files for a given user.

../../_images/92c3cffd25050ab4f251244da26492c7c92b2934aa3ca79dcaf88276e7354117.png

GET /<prefix>/datasets.list

An endpoint to list of all datasets within a given project. This command is equivalent to the renku dataset command.

../../_images/7d07b38317027de1223107c72cfe30b6504da4ec9cd02de273acc1e1700cc262.png

GET /<prefix>/datasets.files_list

An endpoint to list all dataset files within a given project. This command is equivalent to the renku dataset ls-files command.

../../_images/e3bab3fc890f515b43092394a691418b973ec511d7bafd622dce93da376534e5.png

POST /<prefix>/datasets.add

An endpoint to add a file to dataset. This command is equivalent to the renku dataset add command.

The following diagram describes the case when we are adding a file from a local service cache (e.g. user uploaded files):

../../_images/2570e800c3e4a6d8d134861b5764ff8e80a31ef1ec49b67b3789137cfac79476.png

This endpoint also supports adding files to a dataset from an external URL.

../../_images/fd0d0a27bd10fb85890a5c5b72662bf4b9908fd7bfdf311a84e978f1d1b7eb15.png

POST /<prefix>/datasets.create

An endpoint to create a new dataset within a project. This command is equivalent to the renku dataset create command.

../../_images/9093809e7d5200e4d8a609a3be2e7369a0a48da3e3d99e118fea05a271d8f55e.png

POST /<prefix>/datasets.import

An endpoint for importing datasets from external providers. This command is equivalent to the renku dataset import command.

../../_images/0b32bb19b6db198f25710e6a203d70e674a51ef4119d5dd28a8db8946acea18f.png

POST /<prefix>/datasets.edit

An endpoint for editing dataset metadata. This command is equivalent to the renku dataset edit command.

../../_images/9ab79f1729ffbbda9d7415ca18d590f307883974b8b83ebb3a8a2f92c3fb4dbf.png