Core Business Logic

renku.core contains the business logic of renku-python. Functionality is split into subfolders based on topic, such as dataset or workflow.

Command Builder

Most renku commands require context (database/git/etc.) to be set up for them. The command builder pattern makes this easy by wrapping commands in factory methods.

Renku Command Builder .

class renku.command.command_builder.Command

Base renku command builder.

__init__ of Command.

add_injection_pre_hook(order, hook)

Add a pre-execution hook for dependency injection.

Parameters

• order (int) – Determines the order of executed hooks, lower numbers get executed first.

• hook (Callable) – The hook to add.

add_post_hook(order, hook)

Add a post-execution hook.

Parameters

• order (int) – Determines the order of executed hooks, higher numbers get executed first.

• hook (Callable) – The hook to add.

add_pre_hook(order, hook)

Add a pre-execution hook.

Parameters

• order (int) – Determines the order of executed hooks, lower numbers get executed first.

• hook (Callable) – The hook to add.

any_builder_is_instance_of(cls)

Check if any ‘chained’ command builder is an instance of a specific command builder class.

build()

Build (finalize) the command.

Returns

Finalized command that cannot be modified.

Return type

Command

command(operation)

Set the wrapped command.

Parameters

operation (Callable) – The function to wrap in the command builder.

Returns

This command.

Return type

Command

execute(*args, **kwargs)

Execute the wrapped operation.

First executes pre_hooks in ascending order, passing a read/write context between them. It then calls the wrapped operation. The result of the operation then gets pass to all the post_hooks, but in descending order. It then returns the result or error if there was one.

Returns

Result of execution of command.

Return type

CommandResult

property finalized

Whether this builder is still being constructed or has been finalized.

lock_dataset()

Acquire a lock for a dataset.

lock_project()

Acquire a lock for the whole project.

require_clean()

Check that the repository is clean.

require_migration()

Check if a migration is needed.

track_std_streams()

Whether to track STD streams or not.

Returns

This command.

Return type

Command

property will_write_to_database

Will running the command write anything to the metadata store.

with_commit(message=None, commit_if_empty=False, raise_if_empty=False, commit_only=None, skip_staging=False)

Create a commit.

Parameters

• message (str, optional) – The commit message. Auto-generated if left empty (Default value = None).

• commit_if_empty (bool, optional) – Whether to commit if there are no modified files (Default value = False).

• raise_if_empty (bool, optional) – Whether to raise an exception if there are no modified files (Default value = False).

• commit_only (bool, optional) – Only commit the supplied paths (Default value = None).

with_communicator(communicator)

Create a communicator.

Parameters

communicator (CommunicationCallback) – Communicator to use for writing to user.

with_database(write=False, path=None, create=False)

Provide an object database connection.

Parameters

• write (bool, optional) – Whether or not to persist changes to the database (Default value = False).

• path (str, optional) – Location of the database (Default value = None).

• create (bool, optional) – Whether the database should be created if it doesn’t exist (Default value = False).

with_git_isolation()

Whether to run in git isolation or not.

working_directory(directory)

Set the working directory for the command.

Parameters

directory (str) – The working directory to work in.

Returns

This command.

Return type

Command

JSON-LD Schemes

Schema classes used to serialize domain models to JSON-LD.

Activity JSON-LD schema.

class renku.command.schema.activity.ActivitySchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Activity schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.activity.Activity

class renku.command.schema.activity.AssociationSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Association schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.activity.Association

class renku.command.schema.activity.GenerationSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Generation schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.activity.Generation

class renku.command.schema.activity.ParameterValueSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

ParameterValue schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.parameter.ParameterValue

class renku.command.schema.activity.UsageSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Usage schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.activity.Usage

Agents JSON-LD schemes.

class renku.command.schema.agent.PersonSchema(*args, **kwargs)

Bases: calamus.schema.JsonLDSchema

Person schema.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.agent.Person

class renku.command.schema.agent.SoftwareAgentSchema(*args, **kwargs)

Bases: calamus.schema.JsonLDSchema

SoftwareAgent schema.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.agent.SoftwareAgent

Annotation JSON-LD schema.

class renku.command.schema.annotation.AnnotationSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Annotation schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.provenance.annotation.Annotation

Classes for integration with Calamus.

class renku.command.schema.calamus.DateTimeList(*args, **kwargs)

Bases: calamus.fields.DateTime

A DateTime field that might be a list when deserializing.

Create an instance.

class renku.command.schema.calamus.JsonLDSchema(*args, **kwargs)

Bases: calamus.schema.JsonLDSchema

Base schema class for Renku.

Create an instance.

class renku.command.schema.calamus.Nested(*args, **kwargs)

Bases: calamus.fields.Nested

Nested field that passes along commit info.

Init method.

property schema

The nested calamus.Schema object.

This method was copied from marshmallow and modified to support multiple different nested schemes.

class renku.command.schema.calamus.StringList(*args, **kwargs)

Bases: calamus.fields.String

A String field that might be a list when deserializing.

Create an instance.

class renku.command.schema.calamus.Uri(*args, **kwargs)

Bases: calamus.fields._JsonLDField, marshmallow.fields.String, marshmallow.fields.Dict

A Dict/String field.

Create an instance.

Represent a group of run templates.

class renku.command.schema.composite_plan.CompositePlanSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Plan schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.composite_plan.CompositePlan

Datasets JSON-LD schemes.

class renku.command.schema.dataset.DatasetFileSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

DatasetFile schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.DatasetFile

class renku.command.schema.dataset.DatasetSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Dataset schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.Dataset

class renku.command.schema.dataset.DatasetTagSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

DatasetTag schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.DatasetTag

class renku.command.schema.dataset.ImageObjectSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

ImageObject schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.ImageObject

class renku.command.schema.dataset.LanguageSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Language schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.Language

class renku.command.schema.dataset.RemoteEntitySchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

RemoteEntity schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.RemoteEntity

class renku.command.schema.dataset.UrlSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Url schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.dataset.Url

renku.command.schema.dataset.dump_dataset_as_jsonld(dataset)

Return JSON-LD representation of a dataset.

Parameters

dataset (Dataset) – The dataset to convert.

Returns

JSON-LD data of dataset.

Return type

dict

Entities JSON-LD schemes.

class renku.command.schema.entity.CollectionSchema(*args, **kwargs)

Bases: renku.command.schema.entity.EntitySchema

Entity Schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.entity.Collection

class renku.command.schema.entity.EntitySchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Entity Schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.entity.Entity

Parameters JSON-LD schemes.

class renku.command.schema.parameter.CommandInputSchema(*args, **kwargs)

Bases: renku.command.schema.parameter.CommandParameterBaseSchema

CommandInput schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.CommandInput

class renku.command.schema.parameter.CommandOutputSchema(*args, **kwargs)

Bases: renku.command.schema.parameter.CommandParameterBaseSchema

CommandOutput schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.CommandOutput

class renku.command.schema.parameter.CommandParameterBaseSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

CommandParameterBase schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.CommandParameterBase

class renku.command.schema.parameter.CommandParameterSchema(*args, **kwargs)

Bases: renku.command.schema.parameter.CommandParameterBaseSchema

CommandParameter schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.CommandParameter

class renku.command.schema.parameter.MappedIOStreamSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

MappedIOStream schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.MappedIOStream

class renku.command.schema.parameter.ParameterLinkSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

ParameterLink schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.ParameterLink

class renku.command.schema.parameter.ParameterMappingSchema(*args, **kwargs)

Bases: renku.command.schema.parameter.CommandParameterBaseSchema

ParameterMapping schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.parameter.ParameterMapping

Represent run templates.

class renku.command.schema.plan.PlanSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Plan schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.workflow.plan.Plan

Project JSON-LD schema.

class renku.command.schema.project.ProjectSchema(*args, **kwargs)

Bases: renku.command.schema.calamus.JsonLDSchema

Project Schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.domain_model.project.Project

Datasets

Dataset business logic.

renku.core.dataset.dataset.add_datadir_files_to_dataset(dataset)

Add all files in a datasets data directory to the dataset.

Parameters

dataset (Dataset) – The dataset to add data dir files to.

renku.core.dataset.dataset.create_dataset(name, title=None, description=None, creators=None, keywords=None, images=None, update_provenance=True, custom_metadata=None, storage=None, datadir=None)

Create a dataset.

Parameters

• name (str) – Name of the dataset

• title (Optional[str], optional) – Dataset title (Default value = None).

• description (Optional[str], optional) – Dataset description (Default value = None).

• creators (Optional[List[Person]], optional) – Dataset creators (Default value = None).

• keywords (Optional[List[str]], optional) – Dataset keywords (Default value = None).

• images (Optional[List[ImageRequestModel]], optional) – Dataset images (Default value = None).

• update_provenance (bool, optional) – Whether to add this dataset to dataset provenance (Default value = True).

• custom_metadata (Optional[Dict[str, Any]], optional) – Custom JSON-LD metadata (Default value = None).

• storage (Optional[str], optional) – Backend storage’s URI (Default value = None).

• datadir (Optional[Path]) – Dataset’s data directory (Default value = None).

Returns

The created dataset.

Return type

Dataset

renku.core.dataset.dataset.edit_dataset(name, title, description, creators, keywords=<object object>, images=<object object>, custom_metadata=<object object>, custom_metadata_source=<object object>)

Edit dataset metadata.

Parameters

• name (str) – Name of the dataset to edit

• title (Optional[Union[str, NoValueType]]) – New title for the dataset.

• description (Optional[Union[str, NoValueType]]) – New description for the dataset.

• creators (Optional[Union[List[Person], NoValueType]]) – New creators for the dataset.

• keywords (Optional[Union[List[str], NoValueType]]) – New keywords for dataset (Default value = NO_VALUE).

• images (Optional[Union[List[ImageRequestModel], NoValueType]]) – New images for dataset (Default value = NO_VALUE).

• custom_metadata (Optional[Union[Dict, List[Dict], NoValueType]]) – Custom JSON-LD metadata (Default value = NO_VALUE).

• custom_metadata_source (Optional[Union[str, NoValueType]]) – The custom metadata source (Default value = NO_VALUE).

Returns

True if updates were performed.

Return type

bool

renku.core.dataset.dataset.export_dataset(name, provider_name, tag, **kwargs)

Export data to 3rd party provider.

Parameters

• name – Name of dataset to export.

• provider_name – Provider to use for export.

• tag – Dataset tag from which to export.

renku.core.dataset.dataset.file_unlink(name, include, exclude, yes=False)

Remove matching files from a dataset.

Parameters

• name – Dataset name.

• include – Include filter for files.

• exclude – Exclude filter for files.

• yes – Whether to skip user confirmation or not (Default value = False).

Returns

List of files that were removed.

Return type

List[DynamicProxy]

renku.core.dataset.dataset.filter_dataset_files(dataset_gateway, names=None, tag=None, creators=None, include=None, exclude=None, ignore=None, immutable=False, check_data_directory=False)

Filter dataset files by specified filters.

Parameters

• dataset_gateway (IDatasetGateway) – Injected dataset gateway.

• names (Optional[List[str]]) – Filter by specified dataset names (Default value = None).

• tag (Optional[str]) – Filter by specified tag (Default value = None).

• creators (Optional[Union[str, List[str], Tuple[str]]]) – Filter by creators (Default value = None).

• include (Optional[List[str]]) – Tuple containing patterns to which include from result (Default value = None).

• exclude (Optional[List[str]]) – Tuple containing patterns to which exclude from result (Default value = None).

• ignore (Optional[List[str]]) – Ignored datasets (Default value = None).

• immutable (bool) – Return immutable copies of dataset objects (Default value = False).

• check_data_directory (bool) – Whether to check for new files in dataset’s data directory that aren’t in the dataset yet (Default value = False).

Returns

List of filtered files sorted by date added.

Return type

List[DynamicProxy]

renku.core.dataset.dataset.import_dataset(uri, name='', extract=False, yes=False, datadir=None, previous_dataset=None, delete=False, gitlab_token=None, **kwargs)

Import data from a 3rd party provider or another renku project.

Parameters

• uri (str) – DOI or URL of dataset to import.

• name (str) – Name to give imported dataset (Default value = “”).

• extract (bool) – Whether to extract compressed dataset data (Default value = False).

• yes (bool) – Whether to skip user confirmation (Default value = False).

• datadir (Optional[Path]) – Dataset’s data directory (Default value = None).

• previous_dataset (Optional[Dataset]) – Previously imported dataset version (Default value = None).

• delete (bool) – Whether to delete files that don’t exist anymore (Default value = False).

• gitlab_token (Optional[str]) – Gitlab OAuth2 token (Default value = None).

renku.core.dataset.dataset.list_dataset_files(datasets=None, tag=None, creators=None, include=None, exclude=None)

List dataset files.

Parameters

• datasets (List[str]) – Datasets to list files for (Default value = None).

• tag (str) – Tag to filter by (Default value = None).

• creators – Creators to filter by (Default value = None).

• include – Include filters for file paths (Default value = None).

• exclude – Exclude filters for file paths (Default value = None).

Returns

Filtered dataset files.

Return type

List[DynamicProxy]

renku.core.dataset.dataset.list_datasets()

List all datasets.

renku.core.dataset.dataset.mount_external_storage(name, existing, yes, storage_factory)

Mount an external storage to a dataset’s data directory.

Parameters

• name (str) – Name of the dataset

• existing (Optional[Path]) – An existing mount point to use instead of actually mounting the external storage.

• yes (bool) – Don’t prompt when removing non-empty dataset’s data directory.

• storage_factory (IStorageFactory) – Injected storage factory.

renku.core.dataset.dataset.move_files(dataset_gateway, files, to_dataset_name=None)

Move files and their metadata from one or more datasets to a target dataset.

Parameters

• dataset_gateway (IDatasetGateway) – Injected dataset gateway.

• files (Dict[Path, Path]) – Files to move

• to_dataset_name (Optional[str], optional) – Target dataset (Default value = None)

renku.core.dataset.dataset.pull_external_data(name, storage_factory, location=None)

Pull/copy data for an external storage to a dataset’s data directory or a specified location.

Parameters

• name (str) – Name of the dataset

• storage_factory (IStorageFactory) – Injected storage factory.

• location (Optional[Path]) – A directory to copy data to (Default value = None).

renku.core.dataset.dataset.read_dataset_data_location(dataset)

Read data location for a dataset in the config file.

renku.core.dataset.dataset.remove_dataset(name)

Delete a dataset.

Parameters

name – Name of dataset to delete.

renku.core.dataset.dataset.search_datasets(name)

Get all the datasets whose name starts with the given string.

Parameters

name (str) – Beginning of dataset name to search for.

Returns

List of found dataset names.

Return type

List[str]

renku.core.dataset.dataset.set_dataset_images(dataset, images)

Set a dataset’s images.

Parameters

• dataset (Dataset) – The dataset to set images on.

• images (List[ImageRequestModel]) – The images to set.

Returns

True if images were set/modified.

renku.core.dataset.dataset.show_dataset(name, tag=None)

Show detailed dataset information.

Parameters

• name (str) – Name of dataset to show details for.

• tag (str, optional) – Tags for which to get the metadata (Default value = None).

Returns

JSON dictionary of dataset details.

Return type

dict

renku.core.dataset.dataset.store_dataset_data_location(dataset, location)

Store data location for a dataset in the config file.

renku.core.dataset.dataset.unmount_external_storage(name)

Mount an external storage to a dataset’s data directory.

Parameters

name (str) – Name of the dataset

renku.core.dataset.dataset.update_dataset_custom_metadata(dataset, custom_metadata, custom_metadata_source)

Update custom metadata on a dataset.

Parameters

• dataset (Dataset) – The dataset to update.

• custom_metadata (Dict) – Custom JSON-LD metadata to set.

• custom_metadata_source (str) – The source field for the custom metadata.

renku.core.dataset.dataset.update_dataset_git_files(files, ref, delete, dry_run)

Update files and dataset metadata according to their remotes.

Parameters

• files (List[DynamicProxy]) – List of files to be updated.

• ref (Optional[str]) – Reference to use for update.

• delete (bool, optional) – Indicates whether to delete files or not (Default value = False).

• dry_run (bool) – Whether to perform update or only print changes.

Returns

Tuple of updated and deleted file records.

Return type

Tuple[List[DynamicProxy], List[DynamicProxy]]

renku.core.dataset.dataset.update_dataset_local_files(records, check_data_directory)

Update files metadata from the git history.

Parameters

• records (List[DynamicProxy]) – File records to update.

• check_data_directory (bool) – Whether to check the dataset’s data directory for new files.

Returns

Tuple of updated and deleted file records.

Return type

Tuple[List[DynamicProxy], List[DynamicProxy]]

renku.core.dataset.dataset.update_datasets(names, creators, include, exclude, ref, delete, no_external, no_local, no_remote, check_data_directory, update_all, dry_run, dataset_gateway)

Update dataset files.

Parameters

• names (List[str]) – Names of datasets to update.

• creators (Optional[str]) – Creators to filter dataset files by.

• include (Optional[List[str]]) – Include filter for paths to update.

• exclude (Optional[List[str]]) – Exclude filter for paths to update.

• ref (Optional[str]) – Git reference to use for update.

• delete (bool) – Whether to delete files that don’t exist on remote anymore.

• no_external (bool) – Whether to exclude external files from the update.

• no_local (bool) – Whether to exclude local files from the update.

• no_remote (bool) – Whether to exclude remote files from the update.

• check_data_directory (bool) – Whether to check the dataset’s data directory for new files.

• update_all (bool) – Whether to update all datasets.

• dry_run (bool) – Whether to return a preview of what would be updated.

• dataset_gateway (IDatasetGateway) – Injected dataset gateway.

renku.core.dataset.dataset.update_external_files(records, dry_run)

Update files linked to external storage.

Parameters

• records (List[DynamicProxy]) – File records to update.

• dry_run (bool) – Whether to return a preview of what would be updated.

Dataset add business logic.

renku.core.dataset.dataset_add.add_to_dataset(dataset_name, urls, *, importer=None, force=False, create=False, overwrite=False, sources=None, destination='', revision=None, extract=False, clear_files_before=False, total_size=None, datadir=None, storage=None, **kwargs)

Import the data into the data directory.

renku.core.dataset.dataset_add.move_files_to_dataset(files)

Copy/Move files into a dataset’s directory.

Dataset context managers.

class renku.core.dataset.context.DatasetContext(name, create=False, commit_database=False, creator=None, datadir=None, storage=None)

Bases: object

Dataset context manager for metadata changes.

Datasets Provenance.

class renku.core.dataset.datasets_provenance.DatasetsProvenance

Bases: object

A set of datasets.

add_or_update(dataset, date=None, creator=None)

Add/update a dataset according to its new content.

NOTE: This functions always mutates the dataset.

add_tag(dataset, tag)

Add a tag from a dataset.

property datasets

Return an iterator of datasets.

get_all_tags(dataset)

Return the list of all tags for a dataset.

get_by_id(id, immutable=False)

Return a dataset by its id.

get_by_name(name, immutable=False, strict=False)

Return a dataset by its name.

Parameters

• name (str) – Name of the dataset

• immutable (bool) – Whether the dataset will be used as an immutable instance or will be modified (Default value = False).

• strict (bool) – Whether to raise an exception if the dataset doesn’t exist or not (Default value = False)

Returns

Dataset with the specified name if exists.

Return type

Optional[Dataset]

get_previous_version(dataset)

Return the previous version of a dataset if any.

get_provenance_tails()

Return the provenance for all datasets.

remove(dataset, date=None, creator=None)

Remove a dataset.

remove_tag(dataset, tag)

Remove a tag from a dataset.

update_during_migration(dataset, commit_sha, date=None, tags=None, remove=False, replace=False, preserve_identifiers=False)

Add, update, remove, or replace a dataset in migration.

Pointer file business logic.

renku.core.dataset.pointer_file.create_external_file(target, path, checksum=None)

Create a new external file.

renku.core.dataset.pointer_file.create_pointer_file(target, checksum=None)

Create a new pointer file.

renku.core.dataset.pointer_file.get_pointer_file(project_path, path)

Return pointer file from an external file.

renku.core.dataset.pointer_file.is_external_file_updated(project_path, path)

Check if an update to an external file is available.

renku.core.dataset.pointer_file.update_external_file(path, checksum)

Delete existing external file and create a new one.

Renku management dataset request models.

class renku.core.dataset.request_model.ImageRequestModel(content_url, position, mirror_locally=False, safe_image_paths=None)

Bases: object

Model for passing image information to dataset use-cases.

to_image_object(dataset)

Convert request model to ImageObject.

Tag management for dataset.

renku.core.dataset.tag.add_dataset_tag(dataset_name, tag, description='', force=False)

Adds a new tag to a dataset.

Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.

Raises

errors.ParameterError – If tag is too long or contains invalid characters.

renku.core.dataset.tag.get_dataset_by_tag(dataset, tag)

Return a version of dataset that has a specific tag.

Parameters

• dataset (Dataset) – A dataset to return its tagged version.

• tag (str) – Tag name to search for.

Returns

The dataset pointed to by the tag or None if nothing found.

Return type

Optional[Dataset]

renku.core.dataset.tag.list_dataset_tags(dataset_name, format)

List all tags for a dataset.

renku.core.dataset.tag.prompt_access_token(exporter)

Prompt user for an access token for a provider.

Returns

The new access token

renku.core.dataset.tag.prompt_tag_selection(tags)

Prompt user to chose a tag or <HEAD>.

renku.core.dataset.tag.remove_dataset_tags(dataset_name, tags)

Removes tags from a dataset.

Dataset Providers

Providers for dataset import and export

API for providers.

class renku.core.dataset.providers.api.ExporterApi(dataset)

Bases: abc.ABC

Interface defining exporter methods.

property dataset

Return the dataset to be exported.

abstract export(**kwargs)

Execute export process.

abstract get_access_token_url()

Endpoint for creation of access token.

static requires_access_token()

Return if export requires an access token.

abstract set_access_token(access_token)

Set access token.

class renku.core.dataset.providers.api.ImporterApi(uri, original_uri)

Bases: abc.ABC

Interface defining importer methods.

abstract copy_extra_metadata(new_dataset)

Copy provider specific metadata once the dataset is created.

abstract download_files(destination, extract)

Download dataset files from the remote provider.

abstract fetch_provider_dataset()

Deserialize this record to a ProviderDataset.

abstract is_latest_version()

Check if record is at last possible version.

is_version_equal_to(dataset)

Check if a dataset has the same version as the record.

property latest_uri

Get URI of the latest version.

property original_uri

Return original URI of this record without any conversion to DOI.

property provider_dataset

Return the remote dataset. This is only valid after a call to fetch_provider_dataset.

property provider_dataset_files

Return list of dataset files. This is only valid after a call to fetch_provider_dataset.

abstract tag_dataset(name)

Create a tag for the dataset name if the remote dataset has a tag/version.

property uri

Return url of this record.

property version

Get record version.

class renku.core.dataset.providers.api.ProviderApi(uri, **kwargs)

Bases: renku.domain_model.dataset_provider.IDatasetProviderPlugin

Interface defining provider methods.

static add(uri, destination, **kwargs)

Add files from a URI to a dataset.

classmethod dataset_provider()

The definition of the provider.

static get_add_parameters()

Returns parameters that can be set for add.

static get_export_parameters()

Returns parameters that can be set for export.

get_exporter(dataset, *, tag, **kwargs)

Get export manager.

static get_import_parameters()

Returns parameters that can be set for import.

get_importer(**kwargs)

Get import manager.

on_create(dataset)

Hook to perform provider-specific actions on a newly-created dataset.

abstract static supports(uri)

Whether or not this provider supports a given URI.

static supports_add()

Whether this provider supports adding data to datasets.

static supports_create()

Whether this provider supports creating a dataset.

static supports_export()

Whether this provider supports dataset export.

static supports_import()

Whether this provider supports dataset import.

property uri

Return provider’s URI.

class renku.core.dataset.providers.api.ProviderCredentials(provider)

Bases: abc.ABC, collections.UserDict

Credentials of a provider.

NOTE: An empty string, “”, is a valid value. NO_VALUE means that the value for a key is not set.

get_canonical_credentials_names()

Return canonical credentials names that can be used as config keys.

abstract static get_credentials_names()

Return a tuple of the required credentials for a provider.

get_credentials_names_with_no_value()

Return a tuple of credential keys that don’t have a valid value.

get_credentials_section_name()

Get section name for storing credentials.

NOTE: This methods should be overridden by subclasses to allow multiple credentials per providers if needed.

property provider

Return the associated provider instance.

read()

Read credentials from the config and return them. Set non-existing values to None.

store()

Store credentials globally.

class renku.core.dataset.providers.api.ProviderPriority(value)

Bases: enum.IntEnum

Defines the order in which a provider is checked to see if it supports a URI.

Providers that support more specific URIs should have a higher priority so that they are checked first.

Dataverse API integration.

class renku.core.dataset.providers.dataverse.DataverseExporter(*, dataset, server_url=None, dataverse_name=None, publish=False)

Bases: renku.core.dataset.providers.api.ExporterApi

Dataverse export manager.

export(**kwargs)

Execute export process.

get_access_token_url()

Endpoint for creation of access token.

set_access_token(access_token)

Set access token.

class renku.core.dataset.providers.dataverse.DataverseFileSerializer(*, content_size=None, content_url=None, description=None, file_format=None, id=None, identifier=None, name=None, parent_url=None, type=None)

Bases: object

Dataverse record file.

property remote_url

Get remote URL as urllib.ParseResult.

class renku.core.dataset.providers.dataverse.DataverseImporter(uri, original_uri, json)

Bases: renku.core.dataset.providers.repository.RepositoryImporter

Dataverse record serializer.

fetch_provider_dataset()

Deserialize a Dataset.

get_files()

Get Dataverse files metadata as DataverseFileSerializer.

is_latest_version()

Check if record is at last possible version.

property latest_uri

Get URI of latest version.

property version

Get the major and minor version of this dataset.

class renku.core.dataset.providers.dataverse.DataverseProvider(uri, is_doi=False)

Bases: renku.core.dataset.providers.api.ProviderApi

Dataverse API provider.

static get_export_parameters()

Returns parameters that can be set for export.

get_exporter(dataset, *, tag, dataverse_server=None, dataverse_name=None, publish=False, **kwargs)

Create export manager for given dataset.

get_importer(**kwargs)

Get importer for a record from Dataverse.

Returns

The found record

Return type

DataverseImporter

static record_id(uri)

Extract record id from URI.

static supports(uri)

Check if provider supports a given URI.

static supports_export()

Whether this provider supports dataset export.

static supports_import()

Whether this provider supports dataset import.

renku.core.dataset.providers.dataverse.check_dataverse_doi(doi)

Check if a DOI points to a dataverse dataset.

renku.core.dataset.providers.dataverse.check_dataverse_uri(url)

Check if an URL points to a dataverse instance.

renku.core.dataset.providers.dataverse.make_file_url(file_id, base_url)

Create URL to access record by ID.

renku.core.dataset.providers.dataverse.make_records_url(record_id, base_url)

Create URL to access record by ID.

renku.core.dataset.providers.dataverse.make_versions_url(record_id, base_url)

Create URL to access the versions of a record.

Dataverse metadata templates.

DOI API integration.

class renku.core.dataset.providers.doi.DOIImporter(id, doi, url, abstract=None, author=None, categories=None, container_title=None, contributor=None, copyright=None, issued=None, language=None, publisher=None, title=None, type=None, version=None)

Bases: renku.core.dataset.providers.api.ImporterApi

Response from doi.org for DOI metadata.

copy_extra_metadata(new_dataset)

Copy provider specific metadata once the dataset is created.

download_files(destination, extract)

Download dataset files from the remote provider.

fetch_provider_dataset()

Deserialize this record to a ProviderDataset.

is_latest_version()

Check if record is at last possible version.

property latest_uri

Get URI of the latest version.

tag_dataset(name)

Create a tag for the dataset name if the remote dataset has a tag/version.

property version

Get record version.

class renku.core.dataset.providers.doi.DOIProvider(uri, headers=None, timeout=3)

Bases: renku.core.dataset.providers.api.ProviderApi

doi.org registry API provider.

get_importer(**kwargs)

Get import manager.

static supports(uri)

Whether or not this provider supports a given URI.

renku.core.dataset.providers.doi.make_doi_url(doi)

Create URL to access DOI metadata.

Git dataset provider.

class renku.core.dataset.providers.git.GitProvider(uri, **kwargs)

Bases: renku.core.dataset.providers.api.ProviderApi

Git provider.

static add(uri, destination, *, sources=None, revision=None, **kwargs)

Add files from a URI to a dataset.

static get_add_parameters()

Returns parameters that can be set for add.

static supports(uri)

Whether or not this provider supports a given URI.

static supports_add()

Whether this provider supports adding data to datasets.

Local exporter.

class renku.core.dataset.providers.local.FilesystemProvider(uri)

Bases: renku.core.dataset.providers.api.ProviderApi

Local filesystem provider.

static add(uri, destination, *, dataset=None, external=False, move=False, copy=False, link=False, force=False, **kwargs)

Add files from a URI to a dataset.

static get_add_parameters()

Returns parameters that can be set for add.

static get_export_parameters()

Returns parameters that can be set for export.

get_exporter(dataset, *, tag, path=None, **kwargs)

Create export manager for given dataset.

get_importer(uri, **kwargs)

Get import manager.

static supports(uri)

Whether or not this provider supports a given URI.

static supports_add()

Whether this provider supports adding data to datasets.

static supports_export()

Whether this provider supports dataset export.

class renku.core.dataset.providers.local.LocalExporter(dataset, tag, path)

Bases: renku.core.dataset.providers.api.ExporterApi

Local export manager.

export(**kwargs)

Execute entire export process.

get_access_token_url()

Endpoint for creation of access token.

static requires_access_token()

Return if export requires an access token.

set_access_token(access_token)

Set access token.

Models for providers.

class renku.core.dataset.providers.models.DatasetAddAction(value)

Bases: enum.Enum

Types of action when adding a file to a dataset.

class renku.core.dataset.providers.models.DatasetAddMetadata(entity_path, url, action, source, destination, based_on=None, gitignored=False)

Bases: NamedTuple

Metadata for a new file that will be added to a dataset.

Create new instance of DatasetAddMetadata(entity_path, url, action, source, destination, based_on, gitignored)

action

Alias for field number 2

based_on

Alias for field number 5

destination

Alias for field number 4

entity_path

Alias for field number 0

get_absolute_commit_path(project_path)

Return path of the file in the repository.

gitignored

Alias for field number 6

property has_action

Returns if file action is not NONE.

source

Alias for field number 3

url

Alias for field number 1

class renku.core.dataset.providers.models.ProviderDataset(*args, **kwargs)

Bases: renku.domain_model.dataset.Dataset

A Dataset that is imported from a provider.

property files

Return list of existing files.

classmethod from_dataset(dataset)

Create an instance from a Dataset.

classmethod from_jsonld(data, schema_class=None)

Create an instance from JSON-LD data.

property tag

Return dataset’s tag.

class renku.core.dataset.providers.models.ProviderDatasetFile(source, filename, checksum, filesize, filetype, path)

Bases: object

Store metadata for dataset files that will be downloaded from a provider.

class renku.core.dataset.providers.models.ProviderDatasetSchema(*args, **kwargs)

Bases: renku.command.schema.dataset.DatasetSchema

ProviderDataset schema.

Create an instance.

class Meta

Bases: object

Meta class.

model

alias of renku.core.dataset.providers.models.ProviderDataset

class renku.core.dataset.providers.models.ProviderParameter(name, default=None, flags=[], help='', is_flag=False, multiple=False, type=None)

Bases: NamedTuple

Provider-specific parameters.

Create new instance of ProviderParameter(name, default, flags, help, is_flag, multiple, type)

default

Alias for field number 1

flags

Alias for field number 2

help

Alias for field number 3

is_flag

Alias for field number 4

multiple

Alias for field number 5

name

Alias for field number 0

type

Alias for field number 6

OLOS API integration.

class renku.core.dataset.providers.olos.OLOSExporter(*, dataset, server_url=None)

Bases: renku.core.dataset.providers.api.ExporterApi

OLOS export manager.

export(**kwargs)

Execute export process.

get_access_token_url()

Endpoint for creation of access token.

set_access_token(access_token)

Set access token.

class renku.core.dataset.providers.olos.OLOSProvider(uri, is_doi=False)

Bases: renku.core.dataset.providers.api.ProviderApi

Provider for OLOS integration.

static get_export_parameters()

Returns parameters that can be set for export.

get_exporter(dataset, *, tag, dlcm_server=None, **kwargs)

Create export manager for given dataset.

static supports(uri)

Check if provider supports a given URI for importing.

static supports_export()

Whether this provider supports dataset export.

Renku dataset provider.

class renku.core.dataset.providers.renku.RenkuImporter(uri, name, identifier, tag, latest_version_uri, project_url_ssh, project_url_http, gitlab_token, renku_token)

Bases: renku.core.dataset.providers.api.ImporterApi

Renku record serializer.

Create a RenkuImporter from a Dataset.

copy_extra_metadata(new_dataset)

Copy provider specific metadata once the dataset is created.

property datadir_exists

Whether the dataset data directory exists (might be missing in git if empty).

download_files(destination, extract)

Download dataset files from the remote provider.

fetch_provider_dataset()

Return encapsulated dataset instance.

is_latest_version()

Check if dataset is at last possible version.

is_version_equal_to(dataset)

Check if a dataset has the identifier as the record.

property latest_uri

Get URI of the latest version.

property project_url

URL of the Renku project in Gitlab.

property repository

The cloned repository that contains the dataset.

tag_dataset(name)

Create a tag for the dataset name if the remote dataset has a tag/version.

property version

Get record version.

class renku.core.dataset.providers.renku.RenkuProvider(uri, **_)

Bases: renku.core.dataset.providers.api.ProviderApi

Renku API provider.

static get_import_parameters()

Returns parameters that can be set for import.

get_importer(tag=None, gitlab_token=None, **kwargs)

Retrieves a dataset import manager from Renku.

Parameters

• tag (Optional[str]) – Dataset version to import.

• gitlab_token (Optional[str]) – Gitlab access token.

Returns

A Renku import manager.

Return type

RenkuImporter

static supports(uri)

Whether or not this provider supports a given URI.

static supports_import()

Whether this provider supports dataset import.

Base class for online repository data providers.

class renku.core.dataset.providers.repository.RepositoryImporter(uri, original_uri)

Bases: renku.core.dataset.providers.api.ImporterApi, abc.ABC

Online repository importer.

copy_extra_metadata(new_dataset)

Copy provider specific metadata once the dataset is created.

download_files(destination, extract)

Download dataset files from the remote provider.

tag_dataset(name)

Create a tag for the dataset name if the remote dataset has a tag/version.

renku.core.dataset.providers.repository.make_request(url, accept='application/json')

Execute network request.

Web dataset provider.

class renku.core.dataset.providers.web.WebProvider(uri, **kwargs)

Bases: renku.core.dataset.providers.api.ProviderApi

A provider for downloading data from web URLs.

static add(uri, destination, *, extract=False, filename=None, multiple=False, **kwargs)

Add files from a URI to a dataset.

static supports(uri)

Whether or not this provider supports a given URI.

static supports_add()

Whether this provider supports adding data to datasets.

renku.core.dataset.providers.web.download_file(project_path, uri, destination, extract=False, filename=None, multiple=False, delay=0)

Download a file from a URI and return its metadata.

renku.core.dataset.providers.web.download_files(urls, destination, names, extract)

Download multiple files and return their metadata.

Zenodo API integration.

class renku.core.dataset.providers.zenodo.ZenodoDeposition(exporter, id=None)

Bases: object

Zenodo record for a deposit.

attach_metadata(dataset, tag)

Attach metadata to deposition on Zenodo.

property attach_metadata_url

Return URL for attaching metadata.

property deposit_at

Return deposit at URL.

property new_deposit_url

Return URL for creating new deposit.

new_deposition()

Create new deposition on Zenodo.

publish_deposition()

Publish existing deposition.

property publish_url

Returns publish URL.

property published_at

Return published at URL.

upload_file(filepath, path_in_repo)

Upload and attach a file to existing deposition on Zenodo.

property upload_file_url

Return URL for uploading file.

class renku.core.dataset.providers.zenodo.ZenodoExporter(dataset, publish, tag)

Bases: renku.core.dataset.providers.api.ExporterApi

Zenodo export manager.

dataset_to_request()

Prepare dataset metadata for request.

property default_params

Create request default parameters.

export(**kwargs)

Execute entire export process.

get_access_token_url()

Endpoint for creation of access token.

set_access_token(access_token)

Set access token.

property zenodo_url

Returns correct Zenodo URL based on environment.

class renku.core.dataset.providers.zenodo.ZenodoFileSerializer(*, id=None, checksum=None, links=None, filename=None, filesize=None)

Bases: object

Zenodo record file.

property remote_url

Get remote URL as urllib.ParseResult.

property type

Get file type.

class renku.core.dataset.providers.zenodo.ZenodoImporter(*, uri, original_uri, json)

Bases: renku.core.dataset.providers.repository.RepositoryImporter

Zenodo importer.

fetch_provider_dataset()

Deserialize a Dataset.

get_files()

Get Zenodo files metadata as ZenodoFile.

get_jsonld()

Get record metadata as jsonld.

is_latest_version()

Check if this record is the latest version.

property latest_uri

Get URI of latest version.

property version

Get record version.

class renku.core.dataset.providers.zenodo.ZenodoMetadataSerializer(*, access_right=None, communities=None, contributors=None, creators=None, description=None, doi=None, extras=None, grants=None, image_type=None, journal_issue=None, journal_pages=None, journal_title=None, journal_volume=None, keywords=None, language=None, license=None, notes=None, prereserve_doi=None, publication_date=None, publication_type=None, references=None, related_identifiers=None, title=None, upload_type=None, version=None)

Bases: object

Zenodo metadata.

classmethod from_metadata(metadata)

Create an instance from a metadata dict.

Parameters

metadata – The dict data to convert.

Returns

Serializer containing data in deserialized form.

Return type

ZenodoMetadataSerializer

class renku.core.dataset.providers.zenodo.ZenodoProvider(uri, is_doi=False)

Bases: renku.core.dataset.providers.api.ProviderApi

Zenodo registry API provider.

static get_export_parameters()

Returns parameters that can be set for export.

get_exporter(dataset, *, tag, publish=False, **kwargs)

Create export manager for given dataset.

get_importer(**kwargs)

Get importer for a record from Zenodo.

static get_record_id(uri)

Extract record id from URI.

static supports(uri)

Whether or not this provider supports a given URI.

static supports_export()

Whether this provider supports dataset export.

static supports_import()

Whether this provider supports dataset import.

renku.core.dataset.providers.zenodo.make_records_url(record_id)

Create URL to access record by ID.

Parameters

record_id – The id of the record.

Returns

Full URL for the record.

Return type

str

Workflows

Activity management.

renku.core.workflow.activity.add_activity_if_recent(activity, activities)

Add activity to activities if it’s not in the set or is the latest executed instance.

Remove existing activities that were executed earlier.

renku.core.workflow.activity.create_activity_graph(activities, remove_overridden_parents=True, with_inputs_outputs=False)

Create a dependency DAG from activities.

renku.core.workflow.activity.filter_overridden_activities(activities)

Filter out overridden activities from a list of activities.

renku.core.workflow.activity.get_activities_until_paths(paths, sources, activity_gateway, revision=None)

Get all current activities leading to paths, from sources.

renku.core.workflow.activity.get_all_modified_and_deleted_activities_and_entities(repository, activity_gateway)

Return latest activities with at least one modified or deleted input along with the modified/deleted input entity.

An activity can be repeated if more than one of its inputs are modified.

Parameters

• repository – The current Repository.

• activity_gateway (IActivityGateway) – The injected Activity gateway.

Returns

Tuple of modified and deleted

activities and entities.

Return type

Tuple[Set[Tuple[Activity, Entity]], Set[Tuple[Activity, Entity]]]

renku.core.workflow.activity.get_downstream_generating_activities(starting_activities, paths, ignore_deleted, project_path, activity_gateway)

Return activities downstream of passed activities that generate at least a path in paths.

Parameters

• starting_activities (Set[Activity]) – Activities to use as starting/upstream nodes.

• paths (List[str]) – Optional generated paths to end downstream chains at.

• ignore_deleted (bool) – Whether to ignore deleted generations.

• project_path (Path) – Path to project’s root directory.

• activity_gateway (IActivityGateway) – The injected Activity gateway.

Returns

All activities and their downstream activities.

Return type

Set[Activity]

renku.core.workflow.activity.get_latest_activity(activities)

Return the activity that was executed after all other activities.

renku.core.workflow.activity.get_latest_activity_before(activities, activity)

Return the latest activity that was executed before the passed activity.

renku.core.workflow.activity.get_modified_activities(activities, repository)

Get lists of activities that have modified/deleted usage entities.

renku.core.workflow.activity.is_activity_valid(activity)

Return whether this plan has not been deleted.

Parameters

activity (Activity) – The Activity whose Plan should be checked.

Returns

True if the activities’ Plan is still valid, False otherwise.

Return type

bool

renku.core.workflow.activity.revert_activity(*, activity_gateway, activity_id, delete_plan, force, metadata_only)

Revert an activity.

Parameters

• activity_gateway (IActivityGateway) – The injected activity gateway.

• activity_id (str) – ID of the activity to be reverted.

• delete_plan (bool) – Delete the plan if it’s not used by any other activity.

• force (bool) – Revert the activity even if it has some downstream activities.

• metadata_only (bool) – Only revert the metadata and don’t touch generated files.

Returns

The deleted activity.

renku.core.workflow.activity.sort_activities(activities, remove_overridden_parents=True)

Return a sorted list of activities based on their dependencies and execution order.

Plan management.

renku.core.workflow.plan.compose_workflow(name, description, mappings, defaults, links, param_descriptions, map_inputs, map_outputs, map_params, link_all, keywords, steps, sources, sinks, activity_gateway, plan_gateway, project_gateway)

Compose workflows into a CompositePlan.

Parameters

• name (str) – Name of the new composed Plan.

• description (str) – Description for the Plan.

• mappings (List[str]) – Mappings between parameters of this and child Plans.

• defaults (List[str]) – Default values for parameters.

• links (List[str]) – Links between parameters of child Plans.

• param_descriptions (List[str]) – Descriptions of parameters.

• map_inputs (bool) – Whether or not to automatically expose child inputs.

• map_outputs (bool) – Whether or not to automatically expose child outputs.

• map_params (bool) – Whether or not to automatically expose child parameters.

• link_all (bool) – Whether or not to automatically link child steps’ parameters.

• keywords (List[str]) – Keywords for the Plan.

• steps (List[str]) – Child steps to include.

• sources (List[str]) – Starting files when automatically detecting child Plans.

• sinks (List[str]) – Ending files when automatically detecting child Plans.

• activity_gateway (IActivityGateway) – Injected activity gateway.

• plan_gateway (IPlanGateway) – Injected plan gateway.

• project_gateway (IProjectGateway) – Injected project gateway.

Returns

The newly created CompositePlan.

renku.core.workflow.plan.edit_workflow(name, new_name, description, set_params, map_params, rename_params, describe_params, plan_gateway, custom_metadata=None)

Edits a workflow details.

Parameters

• name (str) – Name of the Plan to edit.

• new_name (Optional[str]) – New name of the Plan.

• description (Optional[str]) – New description of the Plan.

• set_params (List[str]) – New default values for parameters.

• map_params (List[str]) – New mappings for Plan.

• rename_params (List[str]) – New names for parameters.

• describe_params (List[str]) – New descriptions for parameters.

• plan_gateway (IPlanGateway) – Injected plan gateway.

• custom_metadata (Dict, optional) – Custom JSON-LD metadata (Default value = None).

Returns

Details of the modified Plan.

renku.core.workflow.plan.export_workflow(name_or_id, plan_gateway, format, output, values)

Export a workflow to a given format.

Parameters

• name_or_id – name or id of the Plan to export

• plan_gateway (IPlanGateway) – The injected Plan gateway.

• format (str) – Format to export to.

• output (Optional[str]) – Output path to store result at.

• values (Optional[str]) – Path to values file to apply before export.

Returns

The exported workflow as string.

renku.core.workflow.plan.get_activities(plan, activity_gateway)

Return all valid activities that use the plan or one of its parent/child derivatives.

renku.core.workflow.plan.get_composite_plans_by_child(plan, plan_gateway)

Return all composite plans that contain a child plan.

renku.core.workflow.plan.get_derivative_chain(plan, plan_gateway)

Return all plans in the derivative chain of a given plan including its parents/children and the plan itself.

renku.core.workflow.plan.get_initial_id(plan, plan_gateway)

Return the id of the first plan in the derivative chain.

renku.core.workflow.plan.get_latest_plan(plan, plan_gateway)

Return the latest version of a given plan in its derivative chain.

renku.core.workflow.plan.is_plan_removed(plan)

Return true if the plan or any plan in its derivative chain is deleted.

renku.core.workflow.plan.list_workflows(plan_gateway, format, columns)

List or manage workflows with subcommands.

Parameters

• plan_gateway (IPlanGateway) – The injected Plan gateway.

• format (str) – The output format.

• columns (List[str]) – The columns to show for tabular output.

Returns

List of workflows formatted by format.

renku.core.workflow.plan.remove_plan(name_or_id, force, plan_gateway, when=datetime.datetime(2022, 11, 10, 14, 10, 2, tzinfo=datetime.timezone(datetime.timedelta(0), 'UTC')))

Remove the workflow by its name or id.

Parameters

• name_or_id (str) – The name of the Plan to remove.

• force (bool) – Whether to force removal or not.

• plan_gateway (IPlanGateway) – The injected Plan gateway.

• when (datetime) – Time of deletion (Default value = current local date/time).

Raises

errors.ParameterError – If the Plan doesn’t exist or was already deleted.

renku.core.workflow.plan.search_workflows(name, plan_gateway)

Get all the workflows whose Plan.name start with the given name.

Parameters

• name (str) – The name to search for.

• plan_gateway (IPlanGateway) – Injected Plan gateway.

Returns

All Plans whose name starts with name.

renku.core.workflow.plan.show_workflow(name_or_id, plan_gateway)

Show the details of a workflow.

Parameters

• name_or_id (str) – Name or id of the Plan to show.

• plan_gateway (IPlanGateway) – The injected Plan gateway.

Returns

Details of the Plan.

renku.core.workflow.plan.visualize_graph(sources, targets, show_files, activity_gateway, revision=None)

Visualize an activity graph.

Parameters

• sources (List[str]) – Input paths to start the visualized graph at.

• targets (List[str]) – Output paths to end the visualized graph at.

• show_files (bool) – Whether or not to show file nodes.

• activity_gateway (IActivityGateway) – The injected activity gateway.

• revision (Optional[str], optional) – Revision or revision range to show the graph for (Default value = None)

Returns

Graph visualization view model.

renku.core.workflow.plan.workflow_inputs(activity_gateway, paths=None)

Get inputs used by workflows.

Parameters

• activity_gateway (IActivityGateway) – The injected activity gateway.

• paths (List[str], optional) – List of paths to consider as inputs (Default value = None).

Returns

Set of input file paths.

Return type

Set[str]

renku.core.workflow.plan.workflow_outputs(activity_gateway, paths=None)

Get inputs used by workflows.

Parameters

• activity_gateway (IActivityGateway) – The injected activity gateway.

• paths (List[str], optional) – List of paths to consider as outputs (Default value = None).

Returns

Set of output file paths.

Return type

Set[str]

Plan execution.

renku.core.workflow.execute.execute_workflow(name_or_id, set_params, provider, config, values, plan_gateway)

Execute a plan with specified values.

Parameters

• name_or_id (str) – Name or id of the Plan to iterate.

• set_params (List[str]) – List of values specified for workflow parameters.

• provider (str) – Name of the workflow provider backend to use for execution.

• config (Optional[str]) – Path to config for the workflow provider.

• values (Optional[str]) – Path to YAMl file containing values specified for workflow parameters.

• plan_gateway (IPlanGateway) – The plan gateway.

renku.core.workflow.execute.execute_workflow_graph(dag, activity_gateway, plan_gateway, provider='cwltool', config=None)

Execute a Run with/without subprocesses.

Parameters

• dag (DiGraph) – The workflow graph to execute.

• activity_gateway (IActivityGateway) – The injected activity gateway.

• plan_gateway (IPlanGateway) – The injected plan gateway.

• provider – Provider to run the workflow with (Default value = “cwltool”).

• config – Path to config for the workflow provider (Default value = None).

renku.core.workflow.execute.iterate_workflow(name_or_id, mapping_path, mappings, dry_run, provider, config, plan_gateway)

Iterate a workflow repeatedly with differing values.

Parameters

• name_or_id (str) – Name or id of the Plan to iterate.

• mapping_path (str) – Path to file defining workflow mappings.

• mappings (List[str]) – List of workflow mappings.

• dry_run (bool) – Whether to preview execution or actually run it.

• provider (str) – Name of the workflow provider backend to use for execution.

• config (Optional[str]) – Path to config for the workflow provider.

• plan_gateway (IPlanGateway) – The plan gateway.

Build an execution graph for a workflow.

class renku.core.workflow.concrete_execution_graph.ExecutionGraph(workflows, virtual_links=False)

Bases: object

Represents an execution graph for one or more workflow steps.

calculate_concrete_execution_graph(virtual_links=False)

Create an execution DAG between Plans showing dependencies between them.

Resolve ParameterLink’s involving ParameterMapping’s to the underlying actual parameters and potentially also virtual links determined by parameter values.

property cycles

Get potential cycles in execution graph.

property workflow_graph

Return a subgraph with only workflows and their dependencies.

Represent a PlanFactory for tracking workflows.

class renku.core.workflow.plan_factory.PlanFactory(command_line, explicit_inputs=None, explicit_outputs=None, explicit_parameters=None, directory=None, working_dir=None, no_input_detection=False, no_output_detection=False, success_codes=None, stdin=None, stdout=None, stderr=None)

Bases: object

Factory for creating a plan from a command line call.

add_command_input(default_value, prefix=None, position=None, postfix=None, name=None, encoding_format=None)

Create a CommandInput.

add_command_output(default_value, prefix=None, position=None, postfix=None, encoding_format=None, name=None, id=None, mapped_to=None)

Create a CommandOutput.

add_command_output_from_input(input, name)

Create a CommandOutput from an input.

add_command_output_from_parameter(parameter, name)

Create a CommandOutput from a parameter.

add_command_parameter(default_value, prefix=None, position=None, name=None)

Create a CommandParameter.

add_explicit_inputs()

Add explicit inputs .

add_explicit_parameters()

Add explicit parameters.

add_indirect_inputs()

Read indirect inputs list and add them to explicit inputs.

add_indirect_outputs()

Read indirect outputs list and add them to explicit outputs.

add_inputs_and_parameters(*arguments)

Yield command input parameters.

add_outputs(candidates)

Yield detected output and changed command input parameter.

get_stream_mapping_for_value(value)

Return a stream mapping if value is a path mapped to a stream.

guess_type(value, ignore_filenames=None)

Return new value and CWL parameter type.

iter_input_files(basedir)

Yield tuples with input id and path.

split_command_and_args()

Return tuple with command and args from command line arguments.

to_plan(project_gateway, name=None, description=None, keywords=None)

Return an instance of Plan based on this factory.

watch(no_output=False)

Watch a Renku repository for changes to detect outputs.

renku.core.workflow.plan_factory.add_indirect_parameter(working_dir, name, value)

Add a parameter to indirect parameters.

renku.core.workflow.plan_factory.add_to_files_list(file_list_path, name, path)

Add a parameter to indirect parameters.

renku.core.workflow.plan_factory.delete_indirect_files_list(working_dir)

Remove indirect inputs, outputs, and parameters list.

renku.core.workflow.plan_factory.get_indirect_inputs_path(project_path)

Return path to file that contains indirect inputs list.

renku.core.workflow.plan_factory.get_indirect_outputs_path(project_path)

Return path to file that contains indirect outputs list.

renku.core.workflow.plan_factory.get_indirect_parameters_path(project_path)

Return path to file that contains indirect parameters list.

renku.core.workflow.plan_factory.read_files_list(files_list)

Read files list yaml containing name:path pairs.

renku.core.workflow.plan_factory.read_indirect_parameters(working_dir)

Read and return indirect parameters.

Represent the Common Workflow Language types.

class renku.core.workflow.types.Directory(path, listing=None)

Represent a directory.

class renku.core.workflow.types.File(path, mime_type=None)

Represent a file.

Resolution of Workflow execution values precedence.

class renku.core.workflow.value_resolution.CompositePlanValueResolver(plan, values=None)

Bases: renku.core.workflow.value_resolution.ValueResolver

Value resolution class for a CompositePlan.

Applies values and default_values to a nested workflow.

Order of precedence is as follows (from lowest to highest): - Default value on a parameter - Default value on a mapping to the parameter - Value passed to a mapping to the parameter - Value passed to the parameter - Value propagated to a parameter from the source of a ParameterLink

apply()

Applies values and default_values to a CompositePlan.

Returns

A CompositePlan with values applied.

class renku.core.workflow.value_resolution.PlanValueResolver(plan, values)

Bases: renku.core.workflow.value_resolution.ValueResolver

Value resolution class for a Plan.

Applies values and default_values to a workflow.

apply()

Applies values and default_values to a Plan.

Returns

A Plan with values applied.

class renku.core.workflow.value_resolution.ValueResolver(plan, values)

Bases: abc.ABC

Value resolution class for an AbstractPlan.

abstract apply()

Applies values and default_values to a potentially nested workflow.

Returns

The AbstractPlan with the user provided values set.

Return type

AbstractPlan

static get(plan, values)

Factory method to obtain the specific ValueResolver for a workflow.

Parameters

• plan (AbstractPlan) – a workflow.

• values (Dict[str, Any]) – user defined dictionary of runtime values for the provided workflow.

Returns

A ValueResolver object.

Return type

“ValueResolver”

renku.core.workflow.value_resolution.apply_parameter_links(workflow)

Apply values from parameter links.

Parameters

workflow (CompositePlan) – The workflow whose links values should be applied on.

Sessions

Docker based interactive session provider.

class renku.core.session.docker.DockerSessionProvider

Bases: renku.domain_model.session.ISessionProvider

A docker based interactive session provider.

build_image(image_descriptor, image_name, config)

Builds the container image.

docker_client()

Get the docker client.

Note

This is not a @property, even though it should be, because pluggy will call it in that case in unrelated parts of the code that will

Raises

errors.DockerError – Exception when docker is not available.

Returns

The docker client.

find_image(image_name, config)

Find the given container image.

get_name()

Return session provider’s name.

session_list(project_name, config)

Lists all the sessions currently running by the given session provider.

Returns

a list of sessions.

Return type

list

session_provider()

Supported session provider.

Returns

a reference to self.

session_start(image_name, project_name, config, cpu_request=None, mem_request=None, disk_request=None, gpu_request=None)

Creates an interactive session.

Returns

a unique id for the created interactive session.

Return type

str

session_stop(project_name, session_name, stop_all)

Stops all or a given interactive session.

session_url(session_name)

Get the URL of the interactive session.

Interactive session business logic.

renku.core.session.session.session_list(config_path, provider=None)

List interactive sessions.

renku.core.session.session.session_open(session_name, provider=None)

Open interactive session in the browser.

renku.core.session.session.session_start(provider, config_path, image_name=None, cpu_request=None, mem_request=None, disk_request=None, gpu_request=None)

Start interactive session.

renku.core.session.session.session_stop(session_name, stop_all=False, provider=None)

Stop interactive session.

Templates

Template management.

class renku.core.template.template.EmbeddedTemplates(path, source, reference, version, skip_validation=False)

Bases: renku.domain_model.template.TemplatesSource

Represent templates that are bundled with Renku.

For embedded templates, source is “renku”. In the old versioning scheme, version is set to the installed Renku version and reference is not set. In the new scheme, both version and reference are set to the template version.

classmethod fetch(source, reference)

Fetch embedded Renku templates.

get_all_references(id)

Return all available references for a template id.

get_latest_reference_and_version(id, reference, version)

Return latest reference and version number of a template.

get_template(id, reference)

Return all available versions for a template id.

class renku.core.template.template.FileAction(value)

Bases: enum.IntEnum

Types of operation when copying a template to a project.

class renku.core.template.template.RepositoryTemplates(path, source, reference, version, repository, skip_validation=False)

Bases: renku.domain_model.template.TemplatesSource

Represent a local/remote template repository.

A template repository is checked out at a specific Git reference if one is provided. However, it’s still possible to get available versions of templates.

classmethod fetch(source, reference)

Fetch a template repository.

get_all_references(id)

Return a list of git tags that are valid SemVer and include a template id.

get_latest_reference_and_version(id, reference, version)

Return latest reference and version number of a template.

get_template(id, reference)

Return a template at a specific reference.

class renku.core.template.template.TemplateAction(value)

Bases: enum.Enum

Types of template rendering.

renku.core.template.template.copy_template_to_project(rendered_template, project, actions, cleanup=True)

Update project files and metadata from a template.

renku.core.template.template.fetch_templates_source(source, reference)

Fetch a template.

renku.core.template.template.get_file_actions(rendered_template, template_action, interactive)

Render a template regarding files in a project.

renku.core.template.template.get_sorted_actions(actions)

Return a sorted actions list.

renku.core.template.template.has_template_checksum()

Return if project has a templates checksum file.

renku.core.template.template.is_renku_template(source)

Return if template comes from Renku.

renku.core.template.template.read_template_checksum()

Read templates checksum file for a project.

renku.core.template.template.set_template_parameters(template, template_metadata, input_parameters, interactive=False)

Set and verify template parameters’ values in the template_metadata.

renku.core.template.template.write_template_checksum(checksums)

Write templates checksum file for a project.

Template use cases.

renku.core.template.usecase.check_for_template_update(project)

Check if the project can be updated to a newer version of the project template.

renku.core.template.usecase.list_templates(source, reference)

Return available templates from a source.

renku.core.template.usecase.select_template(templates_source, id=None)

Select a template from a template source.

renku.core.template.usecase.set_template(source, reference, id, force, interactive, input_parameters, dry_run)

Set template for a project.

renku.core.template.usecase.show_template(source, reference, id)

Show template details.

renku.core.template.usecase.update_template(force, interactive, dry_run)

Update project’s template if possible. Return True if updated.

renku.core.template.usecase.validate_templates(source=None, reference=None)

Validate a template repository.

Parameters

• source (str, optional) – Remote repository URL to clone and check (Default value = None).

• reference (str, optional) – Git commit/branch/tag to check (Default value = None).

Returns

Dictionary containing errors and warnings for manifest and

templates, along with a valid field telling if all checks passed.

Return type

Dict[str, Union[str, Dict[str, List[str]]]]

Errors

Errors that can be raised by renku.core.

Renku exceptions.

exception renku.core.errors.ActivityDownstreamNotEmptyError(activity)

Bases: renku.core.errors.RenkuException

Raised when an activity cannot be deleted because its downstream is not empty.

exception renku.core.errors.AuthenticationError

Bases: renku.core.errors.RenkuException

Raise when there is a problem with authentication.

exception renku.core.errors.ChildWorkflowNotFoundError(child, workflow)

Bases: renku.core.errors.RenkuException

Raised when a parameter reference cannot be resolved to a parameter.

Embed exception and build a custom message.

exception renku.core.errors.CommandFinalizedError

Bases: renku.core.errors.RenkuException

Raised when trying to modify a finalized command builder.

exception renku.core.errors.CommandNotFinalizedError

Bases: renku.core.errors.RenkuException

Raised when a non-finalized command is executed.

exception renku.core.errors.CommitMessageEmpty

Bases: renku.core.errors.RenkuException

Raise invalid commit message.

Build a custom message.

exception renku.core.errors.CommitProcessingError

Bases: renku.core.errors.RenkuException

Raised when a commit couldn’t be processed during graph build.

exception renku.core.errors.ConfigurationError

Bases: renku.core.errors.RenkuException

Raise in case of misconfiguration; use GitConfigurationError for git-related configuration errors.

exception renku.core.errors.DatasetException

Bases: renku.core.errors.RenkuException

Base class for all dataset-related exceptions.

exception renku.core.errors.DatasetExistsError(name)

Bases: renku.core.errors.DatasetException

Raise when trying to create an existing dataset.

exception renku.core.errors.DatasetImageError

Bases: renku.core.errors.DatasetException

Raised when a local dataset image is not accessible.

exception renku.core.errors.DatasetImportError

Bases: renku.core.errors.DatasetException

Raised when a dataset cannot be imported/pulled from a remote source.

exception renku.core.errors.DatasetNotFound(*, name=None, message=None)

Bases: renku.core.errors.DatasetException

Raise when dataset is not found.

Build a custom message.

exception renku.core.errors.DatasetProviderNotFound(*, name=None, uri=None, message=None)

Bases: renku.core.errors.DatasetException, renku.core.errors.ParameterError

Raised when a dataset provider cannot be found based on a URI or a provider name.

exception renku.core.errors.DatasetTagNotFound(tag)

Bases: renku.core.errors.DatasetException

Raise when a tag can’t be found.

exception renku.core.errors.DirectoryNotEmptyError(path)

Bases: renku.core.errors.RenkuException

Raised when a directory passed as output is not empty.

Build a custom message.

exception renku.core.errors.DirtyRenkuDirectory(repository)

Bases: renku.core.errors.RenkuException

Raise when a directory in the renku repository is dirty.

Build a custom message.

exception renku.core.errors.DirtyRepository(repository)

Bases: renku.core.errors.RenkuException

Raise when trying to work with dirty repository.

Build a custom message.

exception renku.core.errors.DockerError(reason)

Bases: renku.core.errors.RenkuException

Raised when error has occurred while executing docker command.

Embed exception and build a custom message.

exception renku.core.errors.DockerfileUpdateError

Bases: renku.core.errors.RenkuException

Raised when the renku version in the Dockerfile couldn’t be updated.

exception renku.core.errors.ExportError

Bases: renku.core.errors.DatasetException

Raised when a dataset cannot be exported.

exception renku.core.errors.ExternalFileNotFound(path)

Bases: renku.core.errors.DatasetException

Raise when an external file is not found.

Build a custom message.

exception renku.core.errors.ExternalStorageDisabled

Bases: renku.core.errors.RenkuException

Raise when disabled repository storage API is trying to be used.

Build a custom message.

exception renku.core.errors.ExternalStorageNotInstalled

Bases: renku.core.errors.RenkuException

Raise when LFS is required but not found or installed in the repository.

Build a custom message.

exception renku.core.errors.FailedMerge(repository, branch, merge_args)

Bases: renku.core.errors.RenkuException

Raise when automatic merge failed.

Build a custom message.

exception renku.core.errors.FileNotFound(path, checksum=None, revision=None)

Bases: renku.core.errors.RenkuException

Raise when a file is not found.

Build a custom message.

exception renku.core.errors.GitCommandError(message='Git command failed.', command=None, stdout=None, stderr=None, status=None)

Bases: renku.core.errors.GitError

Raised when a Git command fails.

Build a custom message.

exception renku.core.errors.GitCommitNotFoundError

Bases: renku.core.errors.GitError

Raised when a commit cannot be found in a Repository.

exception renku.core.errors.GitConfigurationError

Bases: renku.core.errors.GitError

Raised when a git configuration cannot be accessed.

exception renku.core.errors.GitError

Bases: renku.core.errors.RenkuException

Raised when a Git operation fails.

exception renku.core.errors.GitLFSError

Bases: renku.core.errors.RenkuException

Raised when a Git LFS operation fails.

exception renku.core.errors.GitMissingEmail(message=None)

Bases: renku.core.errors.GitConfigurationError

Raise when the email is not configured.

Build a custom message.

exception renku.core.errors.GitMissingUsername(message=None)

Bases: renku.core.errors.GitConfigurationError

Raise when the username is not configured.

Build a custom message.

exception renku.core.errors.GitReferenceNotFoundError

Bases: renku.core.errors.GitError

Raised when a branch or a reference cannot be found.

exception renku.core.errors.GitRemoteNotFoundError

Bases: renku.core.errors.GitError

Raised when a remote cannot be found.

exception renku.core.errors.GraphCycleError(cycles)

Bases: renku.core.errors.RenkuException

Raised when a parameter reference cannot be resolved to a parameter.

Embed exception and build a custom message.

exception renku.core.errors.IncompatibleParametersError(a=None, b=None)

Bases: renku.core.errors.ParameterError

Raise in case of incompatible parameters/flags.

Build a custom message.

exception renku.core.errors.InvalidAccessToken

Bases: renku.core.errors.RenkuException

Raise when access token is incorrect.

Build a custom message.

exception renku.core.errors.InvalidFileOperation

Bases: renku.core.errors.RenkuException

Raise when trying to perform invalid file operation.

exception renku.core.errors.InvalidGitURL

Bases: renku.core.errors.GitError

Raise when a Git URL is not valid.

exception renku.core.errors.InvalidInputPath

Bases: renku.core.errors.RenkuException

Raise when input path does not exist or is not in the repository.

exception renku.core.errors.InvalidOutputPath

Bases: renku.core.errors.RenkuException

Raise when trying to work with an invalid output path.

exception renku.core.errors.InvalidSuccessCode(returncode, success_codes=None)

Bases: renku.core.errors.RenkuException

Raise when the exit-code is not 0 or redefined.

Build a custom message.

exception renku.core.errors.InvalidTemplateError

Bases: renku.core.errors.TemplateError

Raised when using a non-valid template.

exception renku.core.errors.LockError

Bases: renku.core.errors.RenkuException

Raise when a project cannot be locked.

exception renku.core.errors.MappingExistsError(existing_mappings)

Bases: renku.core.errors.RenkuException

Raised when a parameter mapping exists already.

Embed exception and build a custom message.

exception renku.core.errors.MappingNotFoundError(mapping, workflow)

Bases: renku.core.errors.RenkuException

Raised when a parameter mapping does not exist.

Embed exception and build a custom message.

exception renku.core.errors.MetadataMergeError

Bases: renku.core.errors.RenkuException

Raise when merging of metadata failed.

exception renku.core.errors.MigrationError

Bases: renku.core.errors.RenkuException

Raised when something went wrong during migrations.

exception renku.core.errors.MigrationRequired

Bases: renku.core.errors.RenkuException

Raise when migration is required.

Build a custom message.

exception renku.core.errors.MinimumVersionError(current_version, minimum_version)

Bases: renku.core.errors.RenkuException

Raised when accessing a project whose minimum version is larger than the current renku version.

exception renku.core.errors.NodeNotFoundError

Bases: renku.core.errors.RenkuException

Raised when NodeJs is not installed on the system.

Build a custom message.

exception renku.core.errors.NotFound

Bases: renku.core.errors.RenkuException

Raise when an object is not found in KG.

exception renku.core.errors.NotebookSessionImageNotExistError

Bases: renku.core.errors.RenkuException

Raised when a user attempts to start a session with an image that does not exist.

exception renku.core.errors.NotebookSessionNotReadyError

Bases: renku.core.errors.RenkuException

Raised when a user attempts to open a session that is not ready.

exception renku.core.errors.NothingToCommit

Bases: renku.core.errors.RenkuException

Raise when there is nothing to commit.

Build a custom message.

exception renku.core.errors.NothingToExecuteError

Bases: renku.core.errors.RenkuException

Raised when a rerun/update command does not execute any workflows.

exception renku.core.errors.ObjectNotFoundError(filename)

Bases: renku.core.errors.RenkuException

Raised when an object is not found in the storage.

Embed exception and build a custom message.

exception renku.core.errors.OperationError

Bases: renku.core.errors.RenkuException

Raised when an operation at runtime raises an error.

exception renku.core.errors.OutputsNotFound

Bases: renku.core.errors.RenkuException

Raise when there are not any detected outputs in the repository.

Build a custom message.

exception renku.core.errors.ParameterError(message, param_hint=None)

Bases: renku.core.errors.RenkuException

Raise in case of invalid parameter.

Build a custom message.

exception renku.core.errors.ParameterLinkError(reason)

Bases: renku.core.errors.RenkuException

Raised when a parameter link cannot be created.

Embed exception and build a custom message.

exception renku.core.errors.ParameterNotFoundError(parameter, workflow)

Bases: renku.core.errors.RenkuException

Raised when a parameter reference cannot be resolved to a parameter.

Embed exception and build a custom message.

exception renku.core.errors.ProjectNotFound

Bases: renku.core.errors.RenkuException

Raise when one or more projects couldn’t be found in the KG.

exception renku.core.errors.ProjectNotSupported

Bases: renku.core.errors.RenkuException

Raise when project version is newer than the supported version.

Build a custom message.

exception renku.core.errors.ProtectedFiles(ignored)

Bases: renku.core.errors.RenkuException

Raise when trying to work with protected files.

Build a custom message.

exception renku.core.errors.RCloneException

Bases: renku.core.errors.DatasetException

Base class for all rclone-related exceptions.

exception renku.core.errors.RenkuException

Bases: Exception

A base class for all Renku related exception.

You can catch all errors raised by Renku SDK by using except RenkuException:.

exception renku.core.errors.RenkuSaveError

Bases: renku.core.errors.RenkuException

Raised when renku save doesn’t work.

exception renku.core.errors.RenkulabSessionError

Bases: renku.core.errors.RenkuException

Raised when an error occurs trying to start sessions with the notebook service.

exception renku.core.errors.RenkulabSessionGetUrlError

Bases: renku.core.errors.RenkuException

Raised when Renku deployment’s URL cannot be gotten from project’s remotes or configured remotes.

exception renku.core.errors.RequestError

Bases: renku.core.errors.RenkuException

Raise when a requests call fails.

exception renku.core.errors.SHACLValidationError

Bases: renku.core.errors.RenkuException

Raises when SHACL validation of the graph fails.

exception renku.core.errors.StorageObjectNotFound(error=None)

Bases: renku.core.errors.RCloneException

Raised when a file or directory cannot be found in the remote storage.

exception renku.core.errors.StorageProviderNotFound(uri)

Bases: renku.core.errors.DatasetException, renku.core.errors.ParameterError

Raised when a storage provider cannot be found based on a URI.

exception renku.core.errors.TemplateError

Bases: renku.core.errors.RenkuException

Base class for template-related exceptions.

exception renku.core.errors.TemplateMissingReferenceError

Bases: renku.core.errors.TemplateError

Raised when using a non-valid template.

exception renku.core.errors.TemplateNotFoundError

Bases: renku.core.errors.TemplateError

Raised when a template cannot be found in a template source or at a specific reference.

exception renku.core.errors.TemplateUpdateError

Bases: renku.core.errors.TemplateError

Raised when a project couldn’t be updated from its template.

exception renku.core.errors.TerminalSizeError

Bases: renku.core.errors.RenkuException

Raised when terminal is too small for a command.

exception renku.core.errors.UninitializedProject(repo_path)

Bases: renku.core.errors.RenkuException

Raise when a project does not seem to have been initialized yet.

Build a custom message.

exception renku.core.errors.UnmodifiedOutputs(repository, unmodified)

Bases: renku.core.errors.RenkuException

Raise when there are unmodified outputs in the repository.

Build a custom message.

exception renku.core.errors.UsageError

Bases: renku.core.errors.RenkuException

Raise in case of unintended usage of certain function calls.

exception renku.core.errors.WorkflowExecuteError(fail_reason=None)

Bases: renku.core.errors.RenkuException

Raises when a workflow execution fails.

Build a custom message.

exception renku.core.errors.WorkflowRerunError(workflow_file)

Bases: renku.core.errors.RenkuException

Raises when a workflow re-execution fails.

Build a custom message.

Utilities

Communicator classes for printing output.

class renku.core.util.communication.CommunicationCallback

Bases: object

Base communication callback class.

busy(msg)

Indicate a busy status.

For instance, show a spinner in the CLI.

confirm(msg, abort=False, warning=False, default=False)

Get confirmation for an action.

echo(msg, end='\n')

Write a message.

error(msg)

Write an error message.

finalize_progress(name)

End a progress tracker.

has_prompt()

Return True if communicator provides a direct prompt to users.

info(msg)

Write an info message.

prompt(msg, type=None, default=None, **kwargs)

Show a message prompt.

start_progress(name, total, **kwargs)

Create a new progress tracker.

update_progress(name, amount)

Update a progress tracker.

warn(msg)

Write a warning message.

renku.core.util.communication.confirm(msg, abort=False, warning=False, default=False)

Get confirmation for an action from all listeners.

renku.core.util.communication.disable()

Disable all outputs; by default everything is enabled.

renku.core.util.communication.enable()

Enable all outputs.

renku.core.util.communication.error(msg)

Write an error message to all listeners.

renku.core.util.communication.finalize_progress(name)

End a progress tracker on all listeners.

renku.core.util.communication.has_prompt()

Return True if communicator provides a direct prompt to users.

renku.core.util.communication.info(msg)

Write an info message to all listeners.

renku.core.util.communication.prompt(msg, type=None, default=None, **kwargs)

Show a message prompt.

renku.core.util.communication.start_progress(name, total, **kwargs)

Start a progress tracker on all listeners.

renku.core.util.communication.subscribe(listener)

Subscribe a communication listener.

renku.core.util.communication.unsubscribe(listener)

Unsubscribe a communication listener.

renku.core.util.communication.update_progress(name, amount)

Update a progress tracker on all listeners.

renku.core.util.communication.warn(msg)

Write a warning message to all listeners.

Implement various context managers.

class renku.core.util.contexts.Isolation(**kwargs)

Bases: contextlib.ExitStack

Isolate execution.

Create a context manager.

renku.core.util.contexts.Lock(filename, timeout=0, mode='shared', blocking=False)

A file-based lock context manager.

renku.core.util.contexts.chdir(path)

Change the current working directory.

renku.core.util.contexts.measure(message='TOTAL')

Measure execution time of enclosing code block.

class renku.core.util.contexts.redirect_stdin(new_target)

Bases: contextlib.ContextDecorator

Implement missing redirect stdin based on contextlib.py.

Keep the original stream.

renku.core.util.contexts.renku_project_context(path, check_git_path=True)

Provide a project context with repo path injected.

renku.core.util.contexts.wait_for(delay)

Make sure that at least delay seconds are passed during the execution of the wrapped code block.

renku.core.util.contexts.with_project_metadata(project_gateway, database_gateway, read_only=False, name=None, namespace=None, description=None, keywords=None, custom_metadata=None)

Yield an editable metadata object.

Parameters

• project_gateway (IProjectGateway) – Injected project gateway.

• database_gateway (IDatabaseGateway) – Injected database gateway.

• read_only (bool) – Whether to save changes or not (Default value = False).

• name (Optional[str]) – Name of the project (when creating a new one) (Default value = None).

• namespace (Optional[str]) – Namespace of the project (when creating a new one) (Default value = None).

• description (Optional[str]) – Project description (when creating a new one) (Default value = None).

• keywords (Optional[List[str]]) – Keywords for the project (when creating a new one) (Default value = None).

• custom_metadata (Optional[Dict]) – Custom JSON-LD metadata (when creating a new project) (Default value = None).

Renku datetime utilities.

renku.core.util.datetime8601.fix_datetime(value)

Fix timezone of non-aware datetime objects and remove microseconds.

renku.core.util.datetime8601.local_now()

Return current datetime in local timezone.

renku.core.util.datetime8601.match_iso8601(string, pos=0, endpos=9223372036854775807)

Matches zero or more characters at the beginning of the string.

renku.core.util.datetime8601.parse_date(value)

Convert date to datetime.

renku.core.util.datetime8601.validate_iso8601(str_val)

Check if datetime string is in ISO8601 format.

Helper utilities for handling DOIs.

renku.core.util.doi.extract_doi(uri)

Return the DOI in a string if there is one.

renku.core.util.doi.get_doi_url(identifier)

Return DOI URL for a given id.

renku.core.util.doi.is_doi(uri)

Check if URI is DOI.

Helper utilities for handling file size strings.

renku.core.util.file_size.bytes_to_unit(size_in_bytes, unit)

Return size in the provided unit.

renku.core.util.file_size.parse_file_size(size_str)

Parse a human readable file size to bytes.

Git utility functions.

renku.core.util.git.check_global_git_user_is_configured()

Check that git user information is configured.

renku.core.util.git.clone_renku_repository(url, path, gitlab_token=None, deployment_hostname=None, depth=None, install_githooks=False, install_lfs=True, skip_smudge=True, recursive=True, progress=None, config=None, raise_git_except=False, checkout_revision=None, use_renku_credentials=False, reuse_existing_repository=False)

Clone a Renku Repository.

Parameters

• url (str) – The Git URL to clone.

• path (Union[Path, str]) – The path to clone into.

• gitlab_token – The gitlab OAuth2 token (Default value = None).

• deployment_hostname – The hostname of the current renku deployment (Default value = None).

• depth (Optional[int], optional) – The clone depth, number of commits from HEAD (Default value = None).

• install_githooks – Whether to install git hooks (Default value = False).

• install_lfs – Whether to install Git LFS (Default value = True).

• skip_smudge – Whether to pull files from Git LFS (Default value = True).

• recursive – Whether to clone recursively (Default value = True).

• progress – The GitProgress object (Default value = None).

• config (Optional[dict], optional) – Set configuration for the project (Default value = None).

• raise_git_except – Whether to raise git exceptions (Default value = False).

• checkout_revision – The revision to checkout after clone (Default value = None).

• use_renku_credentials (bool, optional) – Whether to use Renku provided credentials (Default value = False).

• reuse_existing_repository (bool, optional) – Whether to clone over an existing repository (Default value = False).

Returns

The cloned repository.

renku.core.util.git.clone_repository(url, path=None, install_githooks=True, install_lfs=True, skip_smudge=True, recursive=True, depth=None, progress=None, config=None, raise_git_except=False, checkout_revision=None, no_checkout=False, clean=False, clone_options=None)

Clone a Git repository and install Git hooks and LFS.

Parameters

• url – The Git URL to clone.

• path (Union[Path, str], optional) – The path to clone into (Default value = None).

• install_githooks – Whether to install git hooks (Default value = True).

• install_lfs – Whether to install Git LFS (Default value = True).

• skip_smudge – Whether to pull files from Git LFS (Default value = True).

• recursive – Whether to clone recursively (Default value = True).

• depth – The clone depth, number of commits from HEAD (Default value = None).

• progress – The GitProgress object (Default value = None).

• config (Optional[dict], optional) – Set configuration for the project (Default value = None).

• raise_git_except – Whether to raise git exceptions (Default value = False).

• checkout_revision – The revision to checkout after clone (Default value = None).

• no_checkout (bool, optional) – Whether to perform a checkout (Default value = False).

• clean (bool, optional) – Whether to require the target folder to be clean (Default value = False).

• clone_options (List[str], optional) – Additional clone options (Default value = None).

Returns

The cloned repository.

renku.core.util.git.commit_changes(*paths, repository, message=None)

Commit paths to the repository.

Parameters

• *paths (Union[Path, str]) – The paths to commit.

• repository (Repository) – The repository to commit to.

• message – The commit message (Default value = None).

Raises

errors.GitError – If paths couldn’t be committed.

Returns

List of paths that were committed.

renku.core.util.git.create_backup_remote(repository, remote_name, url)

Create a backup for remote_name and sets its url to url.

Parameters

• repository (Repository) – The current repository.

• remote_name (str) – The name of the backup remote.

• url (str) – The remote URL.

Returns

Tuple of backup remote name, whether it existed already and the created

remote if successful.

Return type

Tuple[str, bool, Optional[Remote]]

renku.core.util.git.finalize_commit(*, diff_before, repository, transaction_id, commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True, skip_staging=False)

Commit modified/added paths.

renku.core.util.git.get_cache_directory_for_repository(url)

Return a path to project’s cache directory.

Parameters

url – The repository URL.

Returns

The path of the cache.

Return type

Path

renku.core.util.git.get_committer_agent(commit)

Return committer SoftwareAgent.

Parameters

commit (Commit) – The commit to check.

Returns

The agent responsible for the commit.

Return type

SoftwareAgent

renku.core.util.git.get_dirty_paths(repository)

Get paths of dirty files in the repository.

renku.core.util.git.get_entity_from_revision(repository, path, revision=None, bypass_cache=False)

Return an Entity instance from given path and revision.

Parameters

• repository (Repository) – The current repository.

• path (Union[Path, str]) – The path of the entity.

• revision (str, optional) – The revision to check at (Default value = None).

• bypass_cache (bool) – Whether to ignore cached entries and get information from disk (Default value = False).

Returns

The Entity for the given path and revision.

Return type

Entity

renku.core.util.git.get_file_size(repository_path, path)

Return file size for a file inside a git repository.

renku.core.util.git.get_full_repository_path(url)

Extract hostname/path of a git repository from its URL.

Parameters

url (Optional[str]) – The URL.

Returns

The hostname plus path extracted from the URL.

renku.core.util.git.get_git_path(path='.')

Return the repository path.

renku.core.util.git.get_git_progress_instance()

Return a GitProgress object.

renku.core.util.git.get_git_repository(path='.')

Get Git repository from the current path or any of its parents.

Parameters

path – Path to start from (Default value = “.”).

Raises

ValueError – If not inside a git repository.

Returns

Git repository

renku.core.util.git.get_git_user(repository)

Return git user.

Parameters

repository (Optional[Repository]) – The Git repository.

Returns

The person associated with the repository.

Return type

Optional[Person]

renku.core.util.git.get_hook_path(path, name)

Return path to the given named hook in the given repository.

Parameters

• path (Path) – The current Git repository’s path.

• name (str) – The name of the hook.

Returns

Path to the hook.

Return type

Path

renku.core.util.git.get_in_submodules(repository, commit, path)

Resolve filename in submodules.

renku.core.util.git.get_oauth_url(url, gitlab_token)

Format URL with a username and password.

Parameters

• url – The URL to format.

• gitlab_token – The Gitlab OAuth2 Token.

Returns

The URL with credentials added.

renku.core.util.git.get_remote(repository, *, name=None, url=None)

Return repository’s remote using its name or url or return default remote if any.

Parameters

• repository (Optional[Repository]) – The Git repository.

• name (str, optional) – The name of the remote (Default value = None).

• url (str, optional) – The remote URL (Default value = None).

Returns

The remote, if found.

Return type

Optional[Remote]

renku.core.util.git.get_renku_repo_url(remote_url, deployment_hostname=None, access_token=None)

Return a repo url that can be authenticated by renku.

Parameters

• remote_url – The repository URL.

• deployment_hostname – The host name used by this deployment (Default value = None).

• access_token – The OAuth2 access token (Default value = None).

Returns

The Renku repository URL with credentials.

renku.core.util.git.get_repository_name(url)

Extract name of a git repository from its URL.

Parameters

url (str) – The URL to get the repository name from.

Returns

The repository name.

Return type

str

renku.core.util.git.have_same_remote(url1, url2)

Checks if two git urls point to the same remote repo ignoring protocol and credentials.

Parameters

• url1 – The first URL.

• url2 – The second URL.

Returns

True if both URLs point to the same repository.

Return type

bool

renku.core.util.git.is_path_safe(path)

Check if the path should be used in output.

Parameters

path (Union[Path, str]) – The path to check.

Returns

True if the path is safe else False.

Return type

bool

renku.core.util.git.is_valid_git_repository(repository)

Return if is a git repository and has a valid HEAD.

Parameters

repository (Optional[Repository]) – The repository to check.

Returns

Whether or not this is a valid Git repository.

Return type

bool

renku.core.util.git.parse_git_url(url)

Return parsed git url.

Parameters

url (Optional[str]) – The URL to parse.

Raises

errors.InvalidGitURL – If url is empty.

Returns

The parsed GitURL.

Return type

GitURL

renku.core.util.git.prepare_commit(*, repository, commit_only=None, skip_dirty_checks=False, skip_staging=False)

Gather information about repo needed for committing later on.

renku.core.util.git.push_changes(repository, remote=None, reset=True)

Push to a remote branch. If the remote branch is protected a new remote branch will be created and pushed to.

Parameters

• repository (Repository) – The current repository.

• remote (str, optional) – The remote to push to (Default value = None).

• reset (bool, optional) – Whether to reset active branch to its upstream branch, used if changes get pushed to a temporary branch (Default value = True).

Raises

errors.GitError – If there’s no remote or the push fails.

Returns

Name of the branch that was pushed to.

Return type

str

renku.core.util.git.run_command(command, *paths, separator=None, **kwargs)

Execute command by splitting paths to make sure that argument list will be within os limits.

Parameters

• command – A list or tuple containing command and its arguments.

• *paths – Paths to run on.

• separator – Separator for paths if they need to be passed as string. (Default value = None)

Raises

errors.GitError – If a Git subcommand failed.

Returns

Result of last invocation.

renku.core.util.git.shorten_message(message, line_length=100, body_length=65000)

Wraps and shortens a commit message.

Parameters

• message (str) – message to adjust.

• line_length (int, optional) – maximum line length before wrapping. 0 for infinite (Default value = 100).

• body_length (int, optional) – maximum body length before cut. 0 for infinite (Default value = 65000).

Raises

ParameterError – If line_length or body_length < 0

Returns

message wrapped and trimmed.

renku.core.util.git.with_commit(*, repository, transaction_id, commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True, skip_dirty_checks=False)

Automatic commit.

Helpers functions for metadata management/parsing.

renku.core.util.metadata.construct_creator(creator, ignore_email)

Parse input and return an instance of Person.

renku.core.util.metadata.construct_creators(creators, ignore_email=False)

Parse input and return a list of Person.

renku.core.util.metadata.get_canonical_key(key)

Make a consistent configuration key.

renku.core.util.metadata.is_external_file(path, project_path)

Checks if a path is an external file.

renku.core.util.metadata.is_protected_path(path)

Checks if a path is a protected path.

renku.core.util.metadata.make_project_temp_dir(project_path)

Create a temporary directory inside project’s temp path.

renku.core.util.metadata.prompt_for_credentials(provider_credentials)

Prompt for provider credentials if needed and update and store them.

renku.core.util.metadata.read_credentials(section, key)

Read provider’s credentials.

renku.core.util.metadata.read_renku_version_from_dockerfile(path=None)

Read RENKU_VERSION from the content of path if a valid version is available.

renku.core.util.metadata.store_credentials(section, key, value)

Write provider’s credentials.

OS utility functions.

renku.core.util.os.are_paths_related(a, b)

Return True if paths are equal or one is the parent of the other.

renku.core.util.os.create_symlink(path, symlink_path, overwrite=True)

Create a symlink that points from symlink_path to path.

renku.core.util.os.delete_dataset_file(filepath, ignore_errors=True, follow_symlinks=False)

Remove a file/symlink and its pointer file (for external files).

renku.core.util.os.delete_path(path)

Delete a file/directory/symlink.

renku.core.util.os.expand_directories(paths)

Expand directory with all files it contains.

renku.core.util.os.get_absolute_path(path, base=None, resolve_symlinks=False)

Return absolute normalized path.

renku.core.util.os.get_files(path)

Return all files from a starting file/directory.

renku.core.util.os.get_relative_path(path, base, strict=False)

Return a relative path to the base if path is within base without resolving symlinks.

renku.core.util.os.get_relative_path_to_cwd(path)

Get a relative path to current working directory.

renku.core.util.os.get_relative_paths(paths, base)

Return a list of paths relative to a base path.

renku.core.util.os.get_safe_relative_path(path, base)

Return a relative path to the base and check path is within base with all symlinks resolved.

NOTE: This is used to prevent path traversal attack.

renku.core.util.os.hash_file(path, hash_type='sha256')

Calculate the sha256 hash of a file.

renku.core.util.os.hash_file_descriptor(file, hash_type='sha256')

Hash content of a file descriptor.

renku.core.util.os.hash_str(content, hash_type='sha256')

Calculate the sha256 hash of a string.

renku.core.util.os.is_ascii(data)

Check if provided string contains only ascii characters.

renku.core.util.os.is_path_empty(path)

Check if path contains files.

Ref path

target path

renku.core.util.os.is_subpath(path, base)

Return True if path is within base.

renku.core.util.os.matches(path, pattern)

Check if a path matched a given pattern.

renku.core.util.os.normalize_to_ascii(input_string, sep='-')

Convert a string to only contain ASCII characters, with non-ASCII substring replaced with sep.

renku.core.util.os.safe_read_yaml(file)

Parse a YAML file.

Returns

In case of success a dictionary of the YAML’s content, otherwise raises a ParameterError exception.

renku.core.util.os.unmount_path(path)

Unmount the given path and ignore all errors.

Utility for working with HTTP session.

This module provides some wrapper functions around requests library. It sets a timeout and converts exception types whenever needed. Use this module instead of requests.

renku.core.util.requests.check_response(response)

Check for expected response status code.

renku.core.util.requests.delete(url, headers=None)

Send a DELETE request.

renku.core.util.requests.download_file(base_directory, url, filename, extract, chunk_size=16384)

Download a URL to a given location.

renku.core.util.requests.get(url, headers=None, params=None)

Send a GET request.

renku.core.util.requests.get_filename_from_headers(response)

Extract filename from content-disposition headers if available.

renku.core.util.requests.get_redirect_url(url)

Return redirect URL if any; otherwise, return the original URL.

renku.core.util.requests.head(url, *, allow_redirects=False, headers=None)

Send a HEAD request.

renku.core.util.requests.post(url, *, data=None, files=None, headers=None, json=None, params=None)

Send a POST request.

renku.core.util.requests.put(url, *, data=None, files=None, headers=None, params=None)

Send a PUT request.

JSON-LD SHACL validations.

renku.core.util.shacl.validate_graph(graph, shacl_path=None, format='nquads')

Validate the current graph with a SHACL schema.

Uses default schema if not supplied.

Helper utilities for handling URLs.

renku.core.util.urls.get_host(use_project_context=True)

Return the hostname for the resource URIs.

Default is localhost. If RENKU_DOMAIN is set, it overrides the host from remote.

renku.core.util.urls.get_path(url)

Return path part of a url.

renku.core.util.urls.get_scheme(uri)

Return scheme of a URI.

renku.core.util.urls.get_slug(name, invalid_chars=None, lowercase=True)

Create a slug from name.

renku.core.util.urls.is_uri_subfolder(uri, subfolder_uri)

Check if one uri is a ‘subfolder’ of another.

renku.core.util.urls.parse_authentication_endpoint(endpoint=None, use_remote=False)

Return a parsed url.

If an endpoint is provided then use it, otherwise, look for a configured endpoint. If no configured endpoint exists then try to use project’s remote url.

renku.core.util.urls.remove_credentials(url)

Remove username and password from a URL.

renku.core.util.urls.url_to_string(url)

Convert url from list or ParseResult to string.

General utility functions.

renku.core.util.util.NO_VALUE = <object object>

Sentinel to represent a value not being set in cases where None is a valid value.

renku.core.util.util.NoValueType

Type to represent a value not being set in cases where None is a valid value.

alias of object

renku.core.util.util.to_semantic_version(value)

Convert value to SemVer.

renku.core.util.util.to_string(value)

Return a string representation of value and return an empty string for None.

Helper utilities for handling UUIDs.

renku.core.util.uuid.is_uuid(value)

Check if value is UUID4.

Copied from https://stackoverflow.com/questions/19989481/

Support JSON-LD context in models.

class renku.core.util.yaml.NoDatesSafeLoader(stream)

Bases: yaml.cyaml.CSafeLoader

Used to safely load basic python objects but ignore datetime strings.

classmethod remove_implicit_resolver(tag_to_remove)

Remove implicit resolvers for a particular tag.

Takes care not to modify resolvers in super classes.

We want to load datetimes as strings, not dates, because we go on to serialize as json which doesn’t have the advanced types of yaml, and leads to incompatibilities down the track.

renku.core.util.yaml.load_yaml(data)

Load YAML data and return its content as a dict.

renku.core.util.yaml.read_yaml(path)

Load YAML file and return its content as a dict.

renku.core.util.yaml.write_yaml(path, data)

Store data to a YAML file.

Git Internals

Git repository management.

renku.core.git.ensure_clean(ignore_std_streams=False)

Make sure the repository is clean.

renku.core.git.finalize_worktree(isolation, path, branch_name, delete, new_branch, merge_args=('--ff-only',), exception=None)

Cleanup and merge a previously created Git worktree.

renku.core.git.get_mapped_std_streams(lookup_paths, streams=('stdin', 'stdout', 'stderr'))

Get a mapping of standard streams to given paths.

renku.core.git.prepare_worktree(path=None, branch_name=None, commit=None)

Set up a Git worktree to provide isolation.

renku.core.git.with_worktree(path=None, branch_name=None, commit=None, merge_args=('--ff-only',))

Create new worktree.

Git utilities.

class renku.domain_model.git.GitURL(href, path=None, scheme='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, slug=None, regex=None)

Parser for common Git URLs.

Method generated by attrs for class GitURL.

property image

Return image name.

property instance_url

Get the url of the git instance.

classmethod parse(href)

Derive URI components.

renku.domain_model.git.filter_repo_name(repo_name)

Remove the .git extension from the repo name.