Graph services

In Renku, the dependencies of research artifacts are recorded into a knowledge graph. Each project’s local knowledge graph is recorded in its repository; the creation of the global knowledge graph is possible via the graph services. When a project’s repository is pushed to the server, a webhook is triggered that causes the changes represented by the commits and all of the captured dependencies to be rendered as RDF triples and pushed to the triple store.

The graph services are made up of four micro-services: the webhook-service, triples-generator, token-repository and knowledge-graph. The knowledge graph data is stored in the triple store (currently Apache Jena). The basic architecture is illustrated below.

strict digraph architecture { compound=true; newrank=true; graph [fontname="Raleway", nodesep="0.8"]; node [shape="rect", style="filled,rounded", fontname="Raleway"]; edge [fontname="Raleway"] GitLab [fillcolor="lightblue"] UI [fillcolor="#f4d142"] CLI [fillcolor="#f4d142"] WHS [label="Webhook Service" fillcolor="#f4d142"] TG [label="Triples Generator" fillcolor="#f4d142"] KG [label="Knowledge Graph" fillcolor="#f4d142"] Gateway [fillcolor="#f4d142"] Jena [label="Apache Jena" fillcolor="lightblue"] Log [label="Event Log" fillcolor="#f4d142", shape="parallelogram", width=2.0] LogDB [label="Event Log DB" fillcolor="lightblue", shape="parallelogram", width=2.0] subgraph cluster_clients { label="Clients" UI CLI {rank=same; UI, CLI}; } CLI -> GitLab [label=" git push"] WHS -> GitLab [label=" registers webhooks"] GitLab -> WHS [label=" sends Push Event\nwith information about pushed commits"] WHS -> Log [label=" writes Commit Events"] Log -> LogDB [label=" stores Commit Events"] TG -> Log [label=" subscribes for Events"] Log -> TG [label=" pushes Commit Events"] TG -> Jena [label=" generates RDF triples"] KG -> Jena [label=" SPARQL query"] UI -> Gateway [label=" interacts with Graph Services"] Gateway -> WHS [label=" asks to register webhooks,\nchecks Events processing status"] Gateway -> KG [label=" queries for metadata"] }

Sequence diagram of Graph Services APIs and processes.

POST <knowledge-graph>/knowledge-graph/graphql

An endpoint that allows performing GraphQL queries on the Knowledge Graph data.

../../_images/29e1f182646a92a025fa01dc541fc92c99883e34cf082d9043f1b8e5f26400ca.png

POST <webhook-service>/projects/:id/webhooks

An endpoint to create a Graph Services webhook for a project in GitLab.

../../_images/1e077aac7eb84207c4568dff2583ad52ab7a30dbbfd5dc88e1633cd20ed0a97a.png

POST <webhook-service>/projects/:id/webhooks/validation

An endpoint to validate project’s webhook. It checks if a relevant Graph Services webhook exists on the repository in GitLab and if Graph Services have an Access Token associated with the project so they can use it for finding project specific information in GitLab.

../../_images/9151a7137f15294a553788bcc1ce8cc41d47816029522ae1709c2df46d84cf72.png

POST <webhook-service>/webhooks/events

An endpoint to send Push Events containing information about commits pushed to the GitLab.

../../_images/98681136b6d2cfa74c52f44bbf37bef74c8d658b99f43388246781d57358c7e4.png

GET <webhook-service>/projects/:id/events/status

An endpoint that returns information about processing progress of events for a specific project.

../../_images/148204587c565b051e822d339f0b5779e53f52f54576920c717abf194a082e6d.png

Subscription to unprocessed Commit Events

A process initiated and maintained by Triples Generator instances so Event Log can send them Events requiring generation of triples.

../../_images/7fed4858cc61bb9b3bd5ab9b29f560c348cf7652bcf7632661eabf7ccf6e4054.png

Commit Events to RDF Triples

A process responsible for translating unprocessed Commit Events from the Event Log to RDF Triples in the RDF Store. This process runs continuously by polling the Event Log for unprocessed Commit Events.

../../_images/d5cfd4d8d49fc09bc12d35591f05e3a105d844d83c6c94e6093410f63803b768.png

Missed commits synchronization job

A scheduled job which synchronizes state between the Event Log and GitLab and generates Commit Events missing from the Event Log. It runs periodically with a configured interval.

../../_images/3dc2e40e31046440dc1c6b260c0d760a0fedefdb613bca04354efa04c8ca1f2e.png

Knowledge Graph re-provisioning process

A process executed on Triples Generator start-up that checks if triples in the RDF Store were generated with the version of renku-python currently set in the Triples Generator.

../../_images/33d6e47dac30c6a7ab641f130b88d1ff9e3a6466a4227e66c1c03e7951da6a78.png