Core Business Logic¶
renku.core
contains the business logic of renku-python. Functionality is
split into subfolders based on topic, such as dataset
or workflow
.
Command Builder¶
Most renku commands require context (database/git/etc.) to be set up for them. The command builder pattern makes this easy by wrapping commands in factory methods.
Renku Command Builder .
- class renku.command.command_builder.Command[source]¶
Base renku command builder.
__init__ of Command.
- add_injection_pre_hook(order, hook)[source]¶
Add a pre-execution hook for dependency injection.
- Parameters
order (int) – Determines the order of executed hooks, lower numbers get executed first.
hook (Callable) – The hook to add.
- add_post_hook(order, hook)[source]¶
Add a post-execution hook.
- Parameters
order (int) – Determines the order of executed hooks, higher numbers get executed first.
hook (Callable) – The hook to add.
- add_pre_hook(order, hook)[source]¶
Add a pre-execution hook.
- Parameters
order (int) – Determines the order of executed hooks, lower numbers get executed first.
hook (Callable) – The hook to add.
- any_builder_is_instance_of(cls)[source]¶
Check if any ‘chained’ command builder is an instance of a specific command builder class.
- build()[source]¶
Build (finalize) the command.
- Returns
Finalized command that cannot be modified.
- Return type
- command(operation)[source]¶
Set the wrapped command.
- Parameters
operation (Callable) – The function to wrap in the command builder.
- Returns
This command.
- Return type
- execute(*args, **kwargs)[source]¶
Execute the wrapped operation.
First executes pre_hooks in ascending order, passing a read/write context between them. It then calls the wrapped operation. The result of the operation then gets pass to all the post_hooks, but in descending order. It then returns the result or error if there was one.
- Returns
Result of execution of command.
- Return type
CommandResult
- property finalized¶
Whether this builder is still being constructed or has been finalized.
- property will_write_to_database¶
Will running the command write anything to the metadata store.
- with_commit(message=None, commit_if_empty=False, raise_if_empty=False, commit_only=None, skip_staging=False, skip_dirty_checks=False)[source]¶
Create a commit.
- Parameters
message (str, optional) – The commit message. Auto-generated if left empty (Default value = None).
commit_if_empty (bool, optional) – Whether to commit if there are no modified files (Default value = False).
raise_if_empty (bool, optional) – Whether to raise an exception if there are no modified files (Default value = False).
commit_only (bool, optional) – Only commit the supplied paths (Default value = None).
skip_staging (bool) – Don’t commit staged files.
skip_dirty_checks (bool) – Don’t check if paths are dirty or staged.
- with_communicator(communicator)[source]¶
Create a communicator.
- Parameters
communicator (CommunicationCallback) – Communicator to use for writing to user.
- with_database(write=False, path=None, create=False)[source]¶
Provide an object database connection.
- Parameters
write (bool, optional) – Whether or not to persist changes to the database (Default value = False).
path (str, optional) – Location of the database (Default value = None).
create (bool, optional) – Whether the database should be created if it doesn’t exist (Default value = False).
JSON-LD Schemes¶
Schema classes used to serialize domain models to JSON-LD.
Activity JSON-LD schema.
- class renku.command.schema.activity.ActivitySchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Activity schema.
Create an instance.
- class renku.command.schema.activity.AssociationSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Association schema.
Create an instance.
- class renku.command.schema.activity.GenerationSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Generation schema.
Create an instance.
- class renku.command.schema.activity.ParameterValueSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
ParameterValue schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.provenance.parameter.ParameterValue
- class renku.command.schema.activity.UsageSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Usage schema.
Create an instance.
- class renku.command.schema.activity.WorkflowFileActivityCollectionSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
WorkflowFileActivityCollection schema.
Create an instance.
Agents JSON-LD schemes.
- class renku.command.schema.agent.PersonSchema(*args, **kwargs)[source]¶
Bases:
calamus.schema.JsonLDSchema
Person schema.
- class renku.command.schema.agent.SoftwareAgentSchema(*args, **kwargs)[source]¶
Bases:
calamus.schema.JsonLDSchema
SoftwareAgent schema.
Annotation JSON-LD schema.
- class renku.command.schema.annotation.AnnotationSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Annotation schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.provenance.annotation.Annotation
Classes for integration with Calamus.
- class renku.command.schema.calamus.DateTimeList(*args, **kwargs)[source]¶
Bases:
calamus.fields.DateTime
A DateTime field that might be a list when deserializing.
Create an instance.
- class renku.command.schema.calamus.JsonLDSchema(*args, **kwargs)[source]¶
Bases:
calamus.schema.JsonLDSchema
Base schema class for Renku.
Create an instance.
- class renku.command.schema.calamus.Nested(*args, **kwargs)[source]¶
Bases:
calamus.fields.Nested
Nested field that passes along commit info.
Init method.
- property schema¶
The nested
calamus.Schema
object.This method was copied from marshmallow and modified to support multiple different nested schemes.
- class renku.command.schema.calamus.StringList(*args, **kwargs)[source]¶
Bases:
calamus.fields.String
A String field that might be a list when deserializing.
Create an instance.
- class renku.command.schema.calamus.Uri(*args, **kwargs)[source]¶
Bases:
calamus.fields._JsonLDField
,marshmallow.fields.String
,marshmallow.fields.Dict
A Dict/String field.
Create an instance.
Represent a group of run templates.
- class renku.command.schema.composite_plan.CompositePlanSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Plan schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.composite_plan.CompositePlan
Datasets JSON-LD schemes.
- class renku.command.schema.dataset.DatasetFileSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
DatasetFile schema.
Create an instance.
- class renku.command.schema.dataset.DatasetSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Dataset schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.dataset.Dataset
- class renku.command.schema.dataset.DatasetTagSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
DatasetTag schema.
Create an instance.
- class renku.command.schema.dataset.ImageObjectSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
ImageObject schema.
Create an instance.
- class renku.command.schema.dataset.LanguageSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Language schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.dataset.Language
- class renku.command.schema.dataset.RemoteEntitySchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
RemoteEntity schema.
Create an instance.
- class renku.command.schema.dataset.UrlSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Url schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.dataset.Url
- renku.command.schema.dataset.dump_dataset_as_jsonld(dataset)[source]¶
Return JSON-LD representation of a dataset.
- Parameters
dataset (Dataset) – The dataset to convert.
- Returns
JSON-LD data of dataset.
- Return type
dict
Entities JSON-LD schemes.
- class renku.command.schema.entity.CollectionSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.entity.EntitySchema
Entity Schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.entity.Collection
- class renku.command.schema.entity.EntitySchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Entity Schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.entity.Entity
Parameters JSON-LD schemes.
- class renku.command.schema.parameter.CommandInputSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.parameter.CommandParameterBaseSchema
CommandInput schema.
Create an instance.
- class renku.command.schema.parameter.CommandOutputSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.parameter.CommandParameterBaseSchema
CommandOutput schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.CommandOutput
- class renku.command.schema.parameter.CommandParameterBaseSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
CommandParameterBase schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.CommandParameterBase
- class renku.command.schema.parameter.CommandParameterSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.parameter.CommandParameterBaseSchema
CommandParameter schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.CommandParameter
- class renku.command.schema.parameter.MappedIOStreamSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
MappedIOStream schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.MappedIOStream
- class renku.command.schema.parameter.ParameterLinkSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
ParameterLink schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.ParameterLink
- class renku.command.schema.parameter.ParameterMappingSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.parameter.CommandParameterBaseSchema
ParameterMapping schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.parameter.ParameterMapping
Represent run templates.
- class renku.command.schema.plan.PlanSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Plan schema.
Create an instance.
Project JSON-LD schema.
- class renku.command.schema.project.ProjectSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.calamus.JsonLDSchema
Project Schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.project.Project
Represent workflow file run templates.
- class renku.command.schema.workflow_file.WorkflowFileCompositePlanSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.composite_plan.CompositePlanSchema
Plan schema.
Create an instance.
- class renku.command.schema.workflow_file.WorkflowFilePlanSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.plan.PlanSchema
WorkflowFilePlan schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.domain_model.workflow.workflow_file.WorkflowFilePlan
Datasets¶
Dataset business logic.
- renku.core.dataset.dataset.add_datadir_files_to_dataset(dataset)[source]¶
Add all files in a datasets data directory to the dataset.
- Parameters
dataset (Dataset) – The dataset to add data dir files to.
- renku.core.dataset.dataset.create_dataset(name, title=None, description=None, creators=None, keywords=None, images=None, update_provenance=True, custom_metadata=None, storage=None, datadir=None)[source]¶
Create a dataset.
- Parameters
name (str) – Name of the dataset
title (Optional[str], optional) – Dataset title (Default value = None).
description (Optional[str], optional) – Dataset description (Default value = None).
creators (Optional[List[Person]], optional) – Dataset creators (Default value = None).
keywords (Optional[List[str]], optional) – Dataset keywords (Default value = None).
images (Optional[List[ImageRequestModel]], optional) – Dataset images (Default value = None).
update_provenance (bool, optional) – Whether to add this dataset to dataset provenance (Default value = True).
custom_metadata (Optional[Dict[str, Any]], optional) – Custom JSON-LD metadata (Default value = None).
storage (Optional[str], optional) – Backend storage’s URI (Default value = None).
datadir (Optional[Path]) – Dataset’s data directory (Default value = None).
- Returns
The created dataset.
- Return type
- renku.core.dataset.dataset.edit_dataset(name, title, description, creators, keywords=<object object>, images=<object object>, custom_metadata=<object object>, custom_metadata_source=<object object>)[source]¶
Edit dataset metadata.
- Parameters
name (str) – Name of the dataset to edit
title (Optional[Union[str, NoValueType]]) – New title for the dataset.
description (Optional[Union[str, NoValueType]]) – New description for the dataset.
creators (Optional[Union[List[Person], NoValueType]]) – New creators for the dataset.
keywords (Optional[Union[List[str], NoValueType]]) – New keywords for dataset (Default value =
NO_VALUE
).images (Optional[Union[List[ImageRequestModel], NoValueType]]) – New images for dataset (Default value =
NO_VALUE
).custom_metadata (Optional[Union[Dict, List[Dict], NoValueType]]) – Custom JSON-LD metadata (Default value =
NO_VALUE
).custom_metadata_source (Optional[Union[str, NoValueType]]) – The custom metadata source (Default value =
NO_VALUE
).
- Returns
True if updates were performed.
- Return type
bool
- renku.core.dataset.dataset.export_dataset(name, provider_name, tag, **kwargs)[source]¶
Export data to 3rd party provider.
- Parameters
name (str) – Name of dataset to export.
provider_name (str) – Provider to use for export.
tag (str) – Dataset tag from which to export.
- renku.core.dataset.dataset.file_unlink(name, include, exclude, yes=False)[source]¶
Remove matching files from a dataset.
- Parameters
name (str) – Dataset name.
include (Optional[List[str]]) – Include filter for files.
exclude (Optional[List[str]]) – Exclude filter for files.
yes (bool) – Whether to skip user confirmation or not (Default value = False).
- Returns
List of files that were removed.
- Return type
List[DynamicProxy]
- renku.core.dataset.dataset.filter_dataset_files(dataset_gateway, names=None, tag=None, creators=None, include=None, exclude=None, ignore=None, immutable=False, check_data_directory=False)[source]¶
Filter dataset files by specified filters.
- Parameters
dataset_gateway (IDatasetGateway) – Injected dataset gateway.
names (Optional[List[str]]) – Filter by specified dataset names (Default value = None).
tag (Optional[str]) – Filter by specified tag (Default value = None).
creators (Optional[Union[str, List[str], Tuple[str]]]) – Filter by creators (Default value = None).
include (Optional[List[str]]) – Tuple containing patterns to which include from result (Default value = None).
exclude (Optional[List[str]]) – Tuple containing patterns to which exclude from result (Default value = None).
ignore (Optional[List[str]]) – Ignored datasets (Default value = None).
immutable (bool) – Return immutable copies of dataset objects (Default value = False).
check_data_directory (bool) – Whether to check for new files in dataset’s data directory that aren’t in the dataset yet (Default value = False).
- Returns
List of filtered files sorted by date added.
- Return type
List[DynamicProxy]
- renku.core.dataset.dataset.import_dataset(uri, name='', extract=False, yes=False, datadir=None, previous_dataset=None, delete=False, gitlab_token=None, **kwargs)[source]¶
Import data from a 3rd party provider or another renku project.
- Parameters
uri (str) – DOI or URL of dataset to import.
name (str) – Name to give imported dataset (Default value = “”).
extract (bool) – Whether to extract compressed dataset data (Default value = False).
yes (bool) – Whether to skip user confirmation (Default value = False).
datadir (Optional[Path]) – Dataset’s data directory (Default value = None).
previous_dataset (Optional[Dataset]) – Previously imported dataset version (Default value = None).
delete (bool) – Whether to delete files that don’t exist anymore (Default value = False).
gitlab_token (Optional[str]) – Gitlab OAuth2 token (Default value = None).
- renku.core.dataset.dataset.list_dataset_files(datasets=None, tag=None, creators=None, include=None, exclude=None)[source]¶
List dataset files.
- Parameters
datasets (Optional[List[str]]) – Datasets to list files for (Default value = None).
tag (str) – Tag to filter by (Default value = None).
creators (Optional[Union[str, List[str], Tuple[str]]]) – Creators to filter by (Default value = None).
include (Optional[List[str]]) – Include filters for file paths (Default value = None).
exclude (Optional[List[str]]) – Exclude filters for file paths (Default value = None).
- Returns
Filtered dataset files.
- Return type
List[DynamicProxy]
- renku.core.dataset.dataset.mount_external_storage(name, existing, yes)[source]¶
Mount an external storage to a dataset’s data directory.
- Parameters
name (str) – Name of the dataset
existing (Optional[Path]) – An existing mount point to use instead of actually mounting the external storage.
yes (bool) – Don’t prompt when removing non-empty dataset’s data directory.
- renku.core.dataset.dataset.move_files(dataset_gateway, files, to_dataset_name=None)[source]¶
Move files and their metadata from one or more datasets to a target dataset.
- Parameters
dataset_gateway (IDatasetGateway) – Injected dataset gateway.
files (Dict[Path, Path]) – Files to move
to_dataset_name (Optional[str], optional) – Target dataset (Default value = None)
- renku.core.dataset.dataset.pull_external_data(name, location=None)[source]¶
Pull/copy data for an external storage to a dataset’s data directory or a specified location.
- Parameters
name (str) – Name of the dataset
location (Optional[Path]) – A directory to copy data to (Default value = None).
- renku.core.dataset.dataset.read_dataset_data_location(dataset)[source]¶
Read data location for a dataset in the config file.
- renku.core.dataset.dataset.remove_dataset(name)[source]¶
Delete a dataset.
- Parameters
name (str) – Name of dataset to delete.
- renku.core.dataset.dataset.search_datasets(name)[source]¶
Get all the datasets whose name starts with the given string.
- Parameters
name (str) – Beginning of dataset name to search for.
- Returns
List of found dataset names.
- Return type
List[str]
- renku.core.dataset.dataset.set_dataset_images(dataset, images)[source]¶
Set a dataset’s images.
- Parameters
dataset (Dataset) – The dataset to set images on.
images (List[ImageRequestModel]) – The images to set.
- Returns
True if images were set/modified.
- renku.core.dataset.dataset.show_dataset(name, tag=None)[source]¶
Show detailed dataset information.
- Parameters
name (str) – Name of dataset to show details for.
tag (str, optional) – Tags for which to get the metadata (Default value = None).
- Returns
JSON dictionary of dataset details.
- Return type
dict
- renku.core.dataset.dataset.store_dataset_data_location(dataset, location)[source]¶
Store data location for a dataset in the config file.
- renku.core.dataset.dataset.unmount_external_storage(name)[source]¶
Mount an external storage to a dataset’s data directory.
- Parameters
name (str) – Name of the dataset
- renku.core.dataset.dataset.update_dataset_custom_metadata(dataset, custom_metadata, custom_metadata_source)[source]¶
Update custom metadata on a dataset.
- Parameters
dataset (Dataset) – The dataset to update.
custom_metadata (Dict) – Custom JSON-LD metadata to set.
custom_metadata_source (str) – The source field for the custom metadata.
- renku.core.dataset.dataset.update_dataset_git_files(files, ref, delete, dry_run)[source]¶
Update files and dataset metadata according to their remotes.
- Parameters
files (List[DynamicProxy]) – List of files to be updated.
ref (Optional[str]) – Reference to use for update.
delete (bool, optional) – Indicates whether to delete files or not (Default value = False).
dry_run (bool) – Whether to perform update or only print changes.
- Returns
Tuple of updated and deleted file records.
- Return type
Tuple[List[DynamicProxy], List[DynamicProxy]]
- renku.core.dataset.dataset.update_dataset_local_files(records, check_data_directory)[source]¶
Update files metadata from the git history.
- Parameters
records (List[DynamicProxy]) – File records to update.
check_data_directory (bool) – Whether to check the dataset’s data directory for new files.
- Returns
Tuple of updated and deleted file records.
- Return type
Tuple[List[DynamicProxy], List[DynamicProxy]]
- renku.core.dataset.dataset.update_datasets(names, creators, include, exclude, ref, delete, no_external, no_local, no_remote, check_data_directory, update_all, dry_run, plain, dataset_gateway)[source]¶
Update dataset files.
- Parameters
names (List[str]) – Names of datasets to update.
creators (Optional[str]) – Creators to filter dataset files by.
include (Optional[List[str]]) – Include filter for paths to update.
exclude (Optional[List[str]]) – Exclude filter for paths to update.
ref (Optional[str]) – Git reference to use for update.
delete (bool) – Whether to delete files that don’t exist on remote anymore.
no_external (bool) – Whether to exclude external files from the update.
no_local (bool) – Whether to exclude local files from the update.
no_remote (bool) – Whether to exclude remote files from the update.
check_data_directory (bool) – Whether to check the dataset’s data directory for new files.
update_all (bool) – Whether to update all datasets.
dry_run (bool) – Whether to return a preview of what would be updated.
plain (bool) – Whether plain output should be produced.
dataset_gateway (IDatasetGateway) – Injected dataset gateway.
- renku.core.dataset.dataset.update_external_files(records, dry_run)[source]¶
Update files linked to external storage.
- Parameters
records (List[DynamicProxy]) – File records to update.
dry_run (bool) – Whether to return a preview of what would be updated.
Dataset add business logic.
- renku.core.dataset.dataset_add.add_files_to_repository(dataset, files)[source]¶
Track files in project’s repository.
- renku.core.dataset.dataset_add.add_to_dataset(dataset_name, urls, *, importer=None, force=False, create=False, overwrite=False, sources=None, destination='', revision=None, extract=False, clear_files_before=False, total_size=None, datadir=None, storage=None, **kwargs)[source]¶
Import the data into the data directory.
- renku.core.dataset.dataset_add.filter_files(dataset, files, force, overwrite)[source]¶
Filter ignored and overwritten files.
- renku.core.dataset.dataset_add.get_dataset_file_path_within_dataset(dataset, entity_path)[source]¶
Return a dataset file’s path relative to the dataset’s datadir.
NOTE: Dataset files can reside outside the dataset’s datadir.
- renku.core.dataset.dataset_add.get_upload_uri(dataset, entity_path)[source]¶
Return the remote storage path that a dataset file would be located.
- Parameters
dataset (Dataset) – Dataset with a backend storage.
entity_path (Union[Path, str]) – Dataset file’s path (entity path); it is relative to the project’s root.
- Returns
URI within remote storage.
- Return type
str
- renku.core.dataset.dataset_add.move_files_to_dataset(dataset, files)[source]¶
Copy/Move files into a dataset’s directory.
- renku.core.dataset.dataset_add.update_dataset_metadata(dataset, files, clear_files_before)[source]¶
Add newly-added files to the dataset’s metadata.
Dataset context managers.
- class renku.core.dataset.context.DatasetContext(name, create=False, commit_database=False, creator=None, datadir=None, storage=None)[source]¶
Bases:
object
Dataset context manager for metadata changes.
Datasets Provenance.
- class renku.core.dataset.datasets_provenance.DatasetsProvenance[source]¶
Bases:
object
A set of datasets.
- add_or_update(dataset, date=None, creator=None)[source]¶
Add/update a dataset according to its new content.
NOTE: This functions always mutates the dataset.
- property datasets¶
Return an iterator of datasets.
- get_by_name(name, immutable=False, strict=False)[source]¶
Return a dataset by its name.
- Parameters
name (str) – Name of the dataset
immutable (bool) – Whether the dataset will be used as an immutable instance or will be modified (Default value = False).
strict (bool) – Whether to raise an exception if the dataset doesn’t exist or not (Default value = False)
- Returns
Dataset with the specified name if exists.
- Return type
Optional[Dataset]
Pointer file business logic.
- renku.core.dataset.pointer_file.create_external_file(target, path, checksum=None)[source]¶
Create a new external file.
- renku.core.dataset.pointer_file.create_pointer_file(target, checksum=None)[source]¶
Create a new pointer file.
- renku.core.dataset.pointer_file.get_pointer_file(project_path, path)[source]¶
Return pointer file from an external file.
- renku.core.dataset.pointer_file.is_external_file_updated(project_path, path)[source]¶
Check if an update to an external file is available.
- renku.core.dataset.pointer_file.update_external_file(path, checksum)[source]¶
Delete existing external file and create a new one.
Renku management dataset request models.
- class renku.core.dataset.request_model.ImageRequestModel(content_url, position, mirror_locally=False, safe_image_paths=None)[source]¶
Bases:
object
Model for passing image information to dataset use-cases.
Tag management for dataset.
- renku.core.dataset.tag.add_dataset_tag(dataset_name, tag, description='', force=False)[source]¶
Adds a new tag to a dataset.
Validates if the tag already exists and that the tag follows the same rules as docker tags. See https://docs.docker.com/engine/reference/commandline/tag/ for a documentation of docker tag syntax.
- Raises
errors.ParameterError – If tag is too long or contains invalid characters.
- renku.core.dataset.tag.get_dataset_by_tag(dataset, tag)[source]¶
Return a version of dataset that has a specific tag.
- renku.core.dataset.tag.list_dataset_tags(dataset_name, format)[source]¶
List all tags for a dataset.
Dataset Providers¶
Providers for dataset import and export
API for providers.
- class renku.core.dataset.providers.api.AddProviderInterface[source]¶
Bases:
abc.ABC
Interface defining providers that can add data to a dataset.
- class renku.core.dataset.providers.api.ExportProviderInterface[source]¶
Bases:
abc.ABC
Interface defining export providers.
- class renku.core.dataset.providers.api.ExporterApi(dataset)[source]¶
Bases:
abc.ABC
Interface defining exporter methods.
- property dataset¶
Return the dataset to be exported.
- class renku.core.dataset.providers.api.ImportProviderInterface[source]¶
Bases:
abc.ABC
Interface defining import providers.
- class renku.core.dataset.providers.api.ImporterApi(uri, original_uri)[source]¶
Bases:
abc.ABC
Interface defining importer methods.
- abstract copy_extra_metadata(new_dataset)[source]¶
Copy provider specific metadata once the dataset is created.
- abstract download_files(destination, extract)[source]¶
Download dataset files from the remote provider.
- property latest_uri¶
Get URI of the latest version.
- property original_uri¶
Return original URI of this record without any conversion to DOI.
- property provider_dataset¶
Return the remote dataset. This is only valid after a call to
fetch_provider_dataset
.
- property provider_dataset_files¶
Return list of dataset files. This is only valid after a call to
fetch_provider_dataset
.
- abstract tag_dataset(name)[source]¶
Create a tag for the dataset
name
if the remote dataset has a tag/version.
- property uri¶
Return url of this record.
- property version¶
Get record version.
- class renku.core.dataset.providers.api.ProviderApi(uri, **kwargs)[source]¶
Bases:
renku.domain_model.dataset_provider.IDatasetProviderPlugin
Interface defining provider methods.
- property uri¶
Return provider’s URI.
- class renku.core.dataset.providers.api.ProviderCredentials(provider)[source]¶
Bases:
abc.ABC
,collections.UserDict
Credentials of a provider.
NOTE: An empty string, “”, is a valid value.
NO_VALUE
means that the value for a key is not set.- get_canonical_credentials_names()[source]¶
Return canonical credentials names that can be used as config keys.
- get_canonical_credentials_names_with_no_value()[source]¶
Return canonical credentials names that can be used as config keys for keys with no valid value.
- abstract static get_credentials_names()[source]¶
Return a tuple of the required credentials for a provider.
- get_credentials_names_with_no_value()[source]¶
Return a tuple of credential keys that don’t have a valid value.
- get_credentials_section_name()[source]¶
Get section name for storing credentials.
NOTE: This methods should be overridden by subclasses to allow multiple credentials per providers if needed. NOTE: Values used in this method shouldn’t depend on ProviderCredentials attributes since we don’t have those attributes when reading credentials. It’s OK to use ProviderApi attributes.
- property provider¶
Return the associated provider instance.
- class renku.core.dataset.providers.api.StorageProviderInterface[source]¶
Bases:
abc.ABC
Interface defining backend storage providers.
Dataverse API integration.
- class renku.core.dataset.providers.dataverse.DataverseExporter(*, dataset, server_url=None, dataverse_name=None, publish=False)[source]¶
Bases:
renku.core.dataset.providers.api.ExporterApi
Dataverse export manager.
- class renku.core.dataset.providers.dataverse.DataverseFileSerializer(*, content_size=None, content_url=None, description=None, file_format=None, id=None, identifier=None, name=None, parent_url=None, type=None)[source]¶
Bases:
object
Dataverse record file.
- property remote_url¶
Get remote URL as
urllib.ParseResult
.
- class renku.core.dataset.providers.dataverse.DataverseImporter(uri, original_uri, json)[source]¶
Bases:
renku.core.dataset.providers.repository.RepositoryImporter
Dataverse record serializer.
- property latest_uri¶
Get URI of latest version.
- property version¶
Get the major and minor version of this dataset.
- class renku.core.dataset.providers.dataverse.DataverseProvider(uri, is_doi=False)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.ExportProviderInterface
,renku.core.dataset.providers.api.ImportProviderInterface
Dataverse API provider.
- get_exporter(dataset, *, tag, dataverse_server=None, dataverse_name=None, publish=False, **kwargs)[source]¶
Create export manager for given dataset.
- renku.core.dataset.providers.dataverse.check_dataverse_doi(doi)[source]¶
Check if a DOI points to a dataverse dataset.
- renku.core.dataset.providers.dataverse.check_dataverse_uri(url)[source]¶
Check if an URL points to a dataverse instance.
- renku.core.dataset.providers.dataverse.make_file_url(file_id, base_url)[source]¶
Create URL to access record by ID.
- renku.core.dataset.providers.dataverse.make_records_url(record_id, base_url)[source]¶
Create URL to access record by ID.
- renku.core.dataset.providers.dataverse.make_versions_url(record_id, base_url)[source]¶
Create URL to access the versions of a record.
Dataverse metadata templates.
DOI API integration.
- class renku.core.dataset.providers.doi.DOIImporter(id, doi, url, abstract=None, author=None, categories=None, container_title=None, contributor=None, copyright=None, issued=None, language=None, publisher=None, title=None, type=None, version=None)[source]¶
Bases:
renku.core.dataset.providers.api.ImporterApi
Response from doi.org for DOI metadata.
- copy_extra_metadata(new_dataset)[source]¶
Copy provider specific metadata once the dataset is created.
- property latest_uri¶
Get URI of the latest version.
- tag_dataset(name)[source]¶
Create a tag for the dataset
name
if the remote dataset has a tag/version.
- property version¶
Get record version.
- class renku.core.dataset.providers.doi.DOIProvider(uri, headers=None, timeout=3)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.ImportProviderInterface
doi.org registry API provider.
Git dataset provider.
- class renku.core.dataset.providers.git.GitProvider(uri, **kwargs)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.AddProviderInterface
Git provider.
Local exporter.
- class renku.core.dataset.providers.local.FilesystemProvider(uri)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.AddProviderInterface
,renku.core.dataset.providers.api.ExportProviderInterface
Local filesystem provider.
- add(uri, destination, *, external=False, move=False, copy=False, link=False, force=False, **kwargs)[source]¶
Add files from a URI to a dataset.
- class renku.core.dataset.providers.local.LocalExporter(dataset, tag, path)[source]¶
Bases:
renku.core.dataset.providers.api.ExporterApi
Local export manager.
Models for providers.
- class renku.core.dataset.providers.models.DatasetAddAction(value)[source]¶
Bases:
enum.Enum
Types of action when adding a file to a dataset.
- class renku.core.dataset.providers.models.DatasetAddMetadata(entity_path, url, action, source, destination, provider=None, based_on=None)[source]¶
Bases:
object
Metadata for a new file that will be added to a dataset.
- property has_action¶
Returns if file’s action is not NONE.
- property metadata_only¶
Returns if file should be added to a remote storage.
- property remote_storage¶
Returns if file is from a remote storage.
- class renku.core.dataset.providers.models.ProviderDataset(*args, **kwargs)[source]¶
Bases:
renku.domain_model.dataset.Dataset
A Dataset that is imported from a provider.
- property files¶
Return list of existing files.
- property tag¶
Return dataset’s tag.
- class renku.core.dataset.providers.models.ProviderDatasetFile(source, filename, checksum, filesize, filetype, path)[source]¶
Bases:
object
Store metadata for dataset files that will be downloaded from a provider.
- class renku.core.dataset.providers.models.ProviderDatasetSchema(*args, **kwargs)[source]¶
Bases:
renku.command.schema.dataset.DatasetSchema
ProviderDataset schema.
Create an instance.
- class Meta[source]¶
Bases:
object
Meta class.
- model¶
alias of
renku.core.dataset.providers.models.ProviderDataset
- class renku.core.dataset.providers.models.ProviderParameter(name, default=None, flags=[], help='', is_flag=False, multiple=False, type=None)[source]¶
Bases:
NamedTuple
Provider-specific parameters.
Create new instance of ProviderParameter(name, default, flags, help, is_flag, multiple, type)
- default¶
Alias for field number 1
- flags¶
Alias for field number 2
- help¶
Alias for field number 3
- is_flag¶
Alias for field number 4
- multiple¶
Alias for field number 5
- name¶
Alias for field number 0
- type¶
Alias for field number 6
OLOS API integration.
- class renku.core.dataset.providers.olos.OLOSExporter(*, dataset, server_url=None)[source]¶
Bases:
renku.core.dataset.providers.api.ExporterApi
OLOS export manager.
- class renku.core.dataset.providers.olos.OLOSProvider(uri, is_doi=False)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.ExportProviderInterface
Provider for OLOS integration.
Renku dataset provider.
- class renku.core.dataset.providers.renku.RenkuImporter(uri, name, identifier, tag, latest_version_uri, project_url_ssh, project_url_http, gitlab_token, renku_token)[source]¶
Bases:
renku.core.dataset.providers.api.ImporterApi
Renku record serializer.
Create a RenkuImporter from a Dataset.
- copy_extra_metadata(new_dataset)[source]¶
Copy provider specific metadata once the dataset is created.
- property datadir_exists¶
Whether the dataset data directory exists (might be missing in git if empty).
- property latest_uri¶
Get URI of the latest version.
- property project_url¶
URL of the Renku project in Gitlab.
- property repository¶
The cloned repository that contains the dataset.
- tag_dataset(name)[source]¶
Create a tag for the dataset
name
if the remote dataset has a tag/version.
- property version¶
Get record version.
- class renku.core.dataset.providers.renku.RenkuProvider(uri, **_)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.ImportProviderInterface
Renku API provider.
Base class for online repository data providers.
- class renku.core.dataset.providers.repository.RepositoryImporter(uri, original_uri)[source]¶
Bases:
renku.core.dataset.providers.api.ImporterApi
,abc.ABC
Online repository importer.
- renku.core.dataset.providers.repository.make_request(url, accept='application/json')[source]¶
Execute network request.
Web dataset provider.
- class renku.core.dataset.providers.web.WebProvider(uri, **kwargs)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.AddProviderInterface
A provider for downloading data from web URLs.
- renku.core.dataset.providers.web.download_file(project_path, uri, destination, extract=False, filename=None, multiple=False, delay=0)[source]¶
Download a file from a URI and return its metadata.
- renku.core.dataset.providers.web.download_files(urls, destination, names, extract)[source]¶
Download multiple files and return their metadata.
Zenodo API integration.
- class renku.core.dataset.providers.zenodo.ZenodoDeposition(exporter, id=None)[source]¶
Bases:
object
Zenodo record for a deposit.
- property attach_metadata_url¶
Return URL for attaching metadata.
- property deposit_at¶
Return deposit at URL.
- property new_deposit_url¶
Return URL for creating new deposit.
- property publish_url¶
Returns publish URL.
- property published_at¶
Return published at URL.
- upload_file(filepath, path_in_repo)[source]¶
Upload and attach a file to existing deposition on Zenodo.
- property upload_file_url¶
Return URL for uploading file.
- class renku.core.dataset.providers.zenodo.ZenodoExporter(dataset, publish, tag)[source]¶
Bases:
renku.core.dataset.providers.api.ExporterApi
Zenodo export manager.
- property default_params¶
Create request default parameters.
- property zenodo_url¶
Returns correct Zenodo URL based on environment.
- class renku.core.dataset.providers.zenodo.ZenodoFileSerializer(*, id=None, checksum=None, links=None, filename=None, filesize=None)[source]¶
Bases:
object
Zenodo record file.
- property remote_url¶
Get remote URL as
urllib.ParseResult
.
- property type¶
Get file type.
- class renku.core.dataset.providers.zenodo.ZenodoImporter(*, uri, original_uri, json)[source]¶
Bases:
renku.core.dataset.providers.repository.RepositoryImporter
Zenodo importer.
- property latest_uri¶
Get URI of latest version.
- property version¶
Get record version.
- class renku.core.dataset.providers.zenodo.ZenodoMetadataSerializer(*, access_right=None, communities=None, contributors=None, creators=None, description=None, doi=None, extras=None, grants=None, image_type=None, journal_issue=None, journal_pages=None, journal_title=None, journal_volume=None, keywords=None, language=None, license=None, notes=None, prereserve_doi=None, publication_date=None, publication_type=None, references=None, related_identifiers=None, title=None, upload_type=None, version=None)[source]¶
Bases:
object
Zenodo metadata.
- class renku.core.dataset.providers.zenodo.ZenodoProvider(uri, is_doi=False)[source]¶
Bases:
renku.core.dataset.providers.api.ProviderApi
,renku.core.dataset.providers.api.ExportProviderInterface
,renku.core.dataset.providers.api.ImportProviderInterface
Zenodo registry API provider.
Workflows¶
Activity management.
- class renku.core.workflow.activity.ModifiedActivitiesEntities(modified, deleted, hidden_modified)[source]¶
Bases:
NamedTuple
A class containing sets of modified/deleted activities and entities for both normal and hidden entities.
Create new instance of ModifiedActivitiesEntities(modified, deleted, hidden_modified)
- deleted¶
Set of deleted activity and entity tuples.
Set of modified activity and entity tuples for hidden entities.
- modified¶
Set of modified activity and entity tuples.
- renku.core.workflow.activity.add_activity_if_recent(activity, activities)[source]¶
Add
activity
toactivities
if it’s not in the set or is the latest executed instance.Remove existing activities that were executed earlier.
- renku.core.workflow.activity.create_activity_graph(activities, remove_overridden_parents=True, with_inputs_outputs=False, with_hidden_dependencies=False)[source]¶
Create a dependency DAG from activities.
- renku.core.workflow.activity.filter_overridden_activities(activities)[source]¶
Filter out overridden activities from a list of activities.
- renku.core.workflow.activity.get_activities_until_paths(paths, sources, activity_gateway, revision=None)[source]¶
Get all current activities leading to paths, from sources.
- renku.core.workflow.activity.get_all_modified_and_deleted_activities_and_entities(repository, activity_gateway, check_hidden_dependencies=False)[source]¶
Return latest activities with at least one modified or deleted input along with the modified/deleted input entity.
An activity can be repeated if more than one of its inputs are modified.
- Parameters
repository – The current
Repository
.activity_gateway (IActivityGateway) – The injected Activity gateway.
- Returns
Modified and deleted activities and entities.
- Return type
- renku.core.workflow.activity.get_downstream_generating_activities(starting_activities, paths, ignore_deleted, project_path, activity_gateway)[source]¶
Return activities downstream of passed activities that generate at least a path in
paths
.- Parameters
starting_activities (Set[Activity]) – Activities to use as starting/upstream nodes.
paths (List[str]) – Optional generated paths to end downstream chains at.
ignore_deleted (bool) – Whether to ignore deleted generations.
project_path (Path) – Path to project’s root directory.
activity_gateway (IActivityGateway) – The injected Activity gateway.
- Returns
All activities and their downstream activities.
- Return type
Set[Activity]
- renku.core.workflow.activity.get_latest_activity(activities)[source]¶
Return the activity that was executed after all other activities.
- renku.core.workflow.activity.get_latest_activity_before(activities, activity)[source]¶
Return the latest activity that was executed before the passed activity.
- renku.core.workflow.activity.get_modified_activities(activities, repository, check_hidden_dependencies)[source]¶
Get lists of activities that have modified/deleted usage entities.
- renku.core.workflow.activity.is_activity_valid(activity)[source]¶
Return whether this plan has not been deleted.
- Parameters
activity (Activity) – The Activity whose Plan should be checked.
- Returns
True if the activities’ Plan is still valid, False otherwise.
- Return type
bool
- renku.core.workflow.activity.revert_activity(*, activity_gateway, activity_id, delete_plan, force, metadata_only)[source]¶
Revert an activity.
- Parameters
activity_gateway (IActivityGateway) – The injected activity gateway.
activity_id (str) – ID of the activity to be reverted.
delete_plan (bool) – Delete the plan if it’s not used by any other activity.
force (bool) – Revert the activity even if it has some downstream activities.
metadata_only (bool) – Only revert the metadata and don’t touch generated files.
- Returns
The deleted activity.
- renku.core.workflow.activity.sort_activities(activities, remove_overridden_parents=True)[source]¶
Return a sorted list of activities based on their dependencies and execution order.
Plan management.
- renku.core.workflow.plan.compose_workflow(name, description, mappings, defaults, links, param_descriptions, map_inputs, map_outputs, map_params, link_all, keywords, steps, sources, sinks, creators, activity_gateway, plan_gateway, project_gateway)[source]¶
Compose workflows into a CompositePlan.
- Parameters
name (str) – Name of the new composed Plan.
description (Optional[str]) – Description for the Plan.
mappings (Optional[List[str]]) – Mappings between parameters of this and child Plans.
defaults (Optional[List[str]]) – Default values for parameters.
links (Optional[List[str]]) – Links between parameters of child Plans.
param_descriptions (Optional[List[str]]) – Descriptions of parameters.
map_inputs (bool) – Whether or not to automatically expose child inputs.
map_outputs (bool) – Whether or not to automatically expose child outputs.
map_params (bool) – Whether or not to automatically expose child parameters.
link_all (bool) – Whether or not to automatically link child steps’ parameters.
keywords (Optional[List[str]]) – Keywords for the Plan.
steps (Optional[List[str]]) – Child steps to include.
sources (Optional[List[str]]) – Starting files when automatically detecting child Plans.
sinks (Optional[List[str]]) – Ending files when automatically detecting child Plans.
creators (Optional[List[Person]]) – Creator(s) of the composite plan.
activity_gateway (IActivityGateway) – Injected activity gateway.
plan_gateway (IPlanGateway) – Injected plan gateway.
project_gateway (IProjectGateway) – Injected project gateway.
- Returns
The newly created
CompositePlan
.
- renku.core.workflow.plan.edit_workflow(name, new_name, description, set_params, map_params, rename_params, describe_params, creators, keywords, plan_gateway, custom_metadata=None)[source]¶
Edits a workflow details.
- Parameters
name (str) – Name of the Plan to edit.
new_name (Optional[str]) – New name of the Plan.
description (Optional[str]) – New description of the Plan.
set_params (List[str]) – New default values for parameters.
map_params (List[str]) – New mappings for Plan.
rename_params (List[str]) – New names for parameters.
describe_params (List[str]) – New descriptions for parameters.
creators (Union[List[Person], NoValueType]) – Creators of the workflow.
keywords (Union[List[str], NoValueType]) – New keywords for the workflow.
plan_gateway (IPlanGateway) – Injected plan gateway.
custom_metadata (Dict, optional) – Custom JSON-LD metadata (Default value = None).
- Returns
Details of the modified Plan.
- renku.core.workflow.plan.export_workflow(name_or_id, plan_gateway, format, output, values, basedir, resolve_paths, nest_workflows)[source]¶
Export a workflow to a given format.
- Parameters
name_or_id – name or id of the Plan to export
plan_gateway (IPlanGateway) – The injected Plan gateway.
format (str) – Format to export to.
output (Optional[str]) – Output path to store result at.
values (Optional[Dict[str,Any]]) – Parameter names and values to apply before export.
basedir (Optional[str]) – The base path prepended to all paths in the exported workflow, if None it defaults to the absolute path of the renku project.
resolve_paths (Optional[bool]) – Resolve all symlinks and make paths absolute, defaults to True.
nest_workflows (Optional[bool]) – Whether to try to nest all workflows into one specification and file or not, defaults to False.
- Returns
The exported workflow as string.
- renku.core.workflow.plan.get_activities(plan, activity_gateway)[source]¶
Return all valid activities that use the plan or one of its parent/child derivatives.
- renku.core.workflow.plan.get_composite_plans_by_child(plan, plan_gateway)[source]¶
Return all composite plans that contain a child plan.
- renku.core.workflow.plan.get_derivative_chain(plan, plan_gateway)[source]¶
Return all plans in the derivative chain of a given plan including its parents/children and the plan itself.
- renku.core.workflow.plan.get_initial_id(plan, plan_gateway)[source]¶
Return the id of the first plan in the derivative chain.
- renku.core.workflow.plan.get_latest_plan(plan, plan_gateway)[source]¶
Return the latest version of a given plan in its derivative chain.
- renku.core.workflow.plan.get_plan(plan_gateway, name_or_id_or_path=None, workflow_file=None)[source]¶
Return the latest version of a given plan in its derivative chain.
- renku.core.workflow.plan.get_plans_with_metadata(activity_gateway, plan_gateway)[source]¶
Get all plans in the project with additional metadata.
Adds information about last execution, number of executions and whether the plan was used to create files currently existing in the project.
- renku.core.workflow.plan.is_plan_removed(plan)[source]¶
Return true if the plan or any plan in its derivative chain is deleted.
- renku.core.workflow.plan.list_workflows(plan_gateway, format, columns)[source]¶
List or manage workflows with subcommands.
- Parameters
plan_gateway (IPlanGateway) – The injected Plan gateway.
format (str) – The output format.
columns (List[str]) – The columns to show for tabular output.
- Returns
List of workflows formatted by
format
.
- renku.core.workflow.plan.remove_plan(name_or_id, force, plan_gateway)[source]¶
Remove the workflow by its name or id.
- Parameters
name_or_id (str) – The name of the Plan to remove.
force (bool) – Whether to force removal or not.
plan_gateway (IPlanGateway) – The injected Plan gateway.
when (datetime) – Time of deletion (Default value = current local date/time).
- Raises
errors.ParameterError – If the Plan doesn’t exist or was already deleted.
- renku.core.workflow.plan.search_workflows(name, plan_gateway)[source]¶
Get all the workflows whose Plan.name start with the given name.
- Parameters
name (str) – The name to search for.
plan_gateway (IPlanGateway) – Injected Plan gateway.
- Returns
All Plans whose name starts with
name
.
- renku.core.workflow.plan.show_workflow(name_or_id_or_path, activity_gateway, with_metadata=False)[source]¶
Show the details of a workflow.
- Parameters
name_or_id_or_path (str) – Name or id of the Plan to show or path to a workflow file.
activity_gateway (IActivityGateway) – The injected Activity gateway.
with_metadata (bool) – Whether to get additional calculated metadata for the plan.
- Returns
Details of the Plan.
- renku.core.workflow.plan.visualize_graph(sources, targets, show_files, activity_gateway, revision=None)[source]¶
Visualize an activity graph.
- Parameters
sources (List[str]) – Input paths to start the visualized graph at.
targets (List[str]) – Output paths to end the visualized graph at.
show_files (bool) – Whether or not to show file nodes.
activity_gateway (IActivityGateway) – The injected activity gateway.
revision (Optional[str], optional) – Revision or revision range to show the graph for (Default value = None)
- Returns
Graph visualization view model.
- renku.core.workflow.plan.workflow_inputs(activity_gateway, paths=None)[source]¶
Get inputs used by workflows.
- Parameters
activity_gateway (IActivityGateway) – The injected activity gateway.
paths (List[str], optional) – List of paths to consider as inputs (Default value = None).
- Returns
Set of input file paths.
- Return type
Set[str]
- renku.core.workflow.plan.workflow_outputs(activity_gateway, paths=None)[source]¶
Get inputs used by workflows.
- Parameters
activity_gateway (IActivityGateway) – The injected activity gateway.
paths (List[str], optional) – List of paths to consider as outputs (Default value = None).
- Returns
Set of output file paths.
- Return type
Set[str]
Plan execution.
- renku.core.workflow.execute.check_for_cycles(graph)[source]¶
Check for cycles in the graph and raises an error if there are any.
- renku.core.workflow.execute.execute_workflow(name_or_id, set_params, provider, config, values, plan_gateway)[source]¶
Execute a plan with specified values.
- Parameters
name_or_id (str) – Name or id of the Plan to iterate.
set_params (List[str]) – List of values specified for workflow parameters.
provider (str) – Name of the workflow provider backend to use for execution.
config (Optional[str]) – Path to config for the workflow provider.
values (Optional[str]) – Path to YAMl file containing values specified for workflow parameters.
plan_gateway (IPlanGateway) – The plan gateway.
- renku.core.workflow.execute.execute_workflow_graph(dag, activity_gateway, plan_gateway, provider='toil', config=None, workflow_file_plan=None)[source]¶
Execute a Run with/without subprocesses.
- Parameters
dag (DiGraph) – The workflow graph to execute.
activity_gateway (IActivityGateway) – The injected activity gateway.
plan_gateway (IPlanGateway) – The injected plan gateway.
provider – Provider to run the workflow with (Default value = “toil”).
config – Path to config for the workflow provider (Default value = None).
workflow_file_plan (Optional[WorkflowFileCompositePlan) – If passed, a workflow file is executed, so, store related metadata.
- renku.core.workflow.execute.iterate_workflow(name_or_id, mapping_path, mappings, dry_run, provider, config, plan_gateway)[source]¶
Iterate a workflow repeatedly with differing values.
- Parameters
name_or_id (str) – Name or id of the Plan to iterate.
mapping_path (str) – Path to file defining workflow mappings.
mappings (List[str]) – List of workflow mappings.
dry_run (bool) – Whether to preview execution or actually run it.
provider (str) – Name of the workflow provider backend to use for execution.
config (Optional[str]) – Path to config for the workflow provider.
plan_gateway (IPlanGateway) – The plan gateway.
Build an execution graph for a workflow.
- class renku.core.workflow.model.concrete_execution_graph.ExecutionGraph(workflows, virtual_links=False)[source]¶
Bases:
object
Represents an execution graph for one or more workflow steps.
- static are_paths_linked(*, output, input)[source]¶
Return True if input has a relation to output (i.e. can be generated by it).
- calculate_concrete_execution_graph(virtual_links=False)[source]¶
Create an execution DAG between Plans showing dependencies between them.
Resolve ParameterLink’s involving ParameterMapping’s to the underlying actual parameters and potentially also virtual links determined by parameter values.
- property cycles¶
Get potential cycles in execution graph.
- property workflow_graph¶
Return a subgraph with only workflows and their dependencies.
Models to represent a workflow definition file.
- class renku.core.workflow.model.workflow_file.BaseParameter(description=None, implicit=False, name=None, name_set_by_user=None, position=None, prefix=None)[source]¶
Bases:
object
Base class for Input, Output, and Parameter.
- class renku.core.workflow.model.workflow_file.BasePath(path, mapped_to=None, persist=True, **kwargs)[source]¶
Bases:
renku.core.workflow.model.workflow_file.BaseParameter
Base for workflow Input/Output.
- class renku.core.workflow.model.workflow_file.HiddenParameter(value, **kwargs)[source]¶
Bases:
renku.core.workflow.model.workflow_file.Parameter
A parameter that isn’t defined by user and is created by Renku.
- class renku.core.workflow.model.workflow_file.Input(path, mapped_to=None, persist=True, **kwargs)[source]¶
Bases:
renku.core.workflow.model.workflow_file.BasePath
An input to a workflow file.
- class renku.core.workflow.model.workflow_file.Output(path, mapped_to=None, persist=True, **kwargs)[source]¶
Bases:
renku.core.workflow.model.workflow_file.BasePath
An output from a workflow file.
- class renku.core.workflow.model.workflow_file.Parameter(value, **kwargs)[source]¶
Bases:
renku.core.workflow.model.workflow_file.BaseParameter
A parameter for a workflow file.
- class renku.core.workflow.model.workflow_file.Step(*, command, date_created=None, description=None, inputs=None, keywords=None, name, original_command=None, outputs=None, parameters=None, path, success_codes=None, workflow_file_name)[source]¶
Bases:
object
A single step in a workflow file.
- class renku.core.workflow.model.workflow_file.StepCommandParser(step)[source]¶
Bases:
bashlex.ast.nodevisitor
A class to parse a workflow file step’s command and figure out position of its arguments.
- find_and_process_argument(*, cls, collection, prefix, value, exclude_prefix, are_values_equal)[source]¶
Find the given input/output/parameter in the collection.
- find_argument(prefix, value, search_outputs, exclude_prefix)[source]¶
Check the argument against inputs/outputs/parameters to find its type.
- Parameters
prefix (Optional[str]) – Argument prefix.
value (str) – Argument value.
search_outputs (bool) – Whether to search outputs or not. When no part of a command hasn’t found yet, we shouldn’t search outputs.
exclude_prefix (bool) – Whether prefix must be excluded in search or not. When the command hasn’t been fully found, the prefix might be part of the command and not the argument.
- Returns
An Input, Output, or Parameter instance if a match is found; None otherwise.
- Return type
Optional[BaseParameter]
- find_argument_by_name(name, collection=None)[source]¶
Find an inputs/outputs/parameters with the given name.
- find_parameter(prefix, value, exclude_prefix)[source]¶
Find a parameter based on its value and prefix; adjust parameter’s prefix if needed.
- parse_command()[source]¶
Parse the command and assign some properties (e.g. position) for parameters and the command.
- class renku.core.workflow.model.workflow_file.WorkflowFile(path, steps, name, description=None, keywords=None)[source]¶
Bases:
object
A workflow definition file.
- renku.core.workflow.model.workflow_file.calculate_derivatives(plan, plan_gateway, sequence=None)[source]¶
Check for existing plans and set derivative chain if needed.
- Parameters
plan (Union[WorkflowFileCompositePlan, WorkflowFilePlan]) – The potential derivative plan.
plan_gateway (IPlanGateway) –
PlanGateway
instance.sequence (Optional[itertools.count]) – sequence is used to generate deterministic IDs in case a previous ID cannot be used for a plan (e.g. when a plan was deleted and re-executed) (Default value = False).
- Returns
- An existing plan, if found and is the same; otherwise, the
passed-in
plan
.
- Return type
- renku.core.workflow.model.workflow_file.generate_qualified_plan_name(workflow_file_name, step_name)[source]¶
Generate name for
WorkflowFile
andStep
.
Represent a PlanFactory
for tracking workflows.
- class renku.core.workflow.plan_factory.PlanFactory(command_line, explicit_inputs=None, explicit_outputs=None, explicit_parameters=None, directory=None, working_dir=None, no_input_detection=False, no_output_detection=False, success_codes=None, stdin=None, stdout=None, stderr=None)[source]¶
Bases:
object
Factory for creating a plan from a command line call.
- add_command_input(default_value, prefix=None, position=None, postfix=None, name=None, encoding_format=None)[source]¶
Create a CommandInput.
- add_command_output(default_value, prefix=None, position=None, postfix=None, encoding_format=None, name=None, id=None, mapped_to=None)[source]¶
Create a CommandOutput.
- add_command_output_from_parameter(parameter, name)[source]¶
Create a CommandOutput from a parameter.
- add_command_parameter(default_value, prefix=None, position=None, name=None)[source]¶
Create a CommandParameter.
- get_stream_mapping_for_value(value)[source]¶
Return a stream mapping if value is a path mapped to a stream.
- renku.core.workflow.plan_factory.add_indirect_parameter(working_dir, name, value)[source]¶
Add a parameter to indirect parameters.
- renku.core.workflow.plan_factory.add_to_files_list(file_list_path, name, path)[source]¶
Add a parameter to indirect parameters.
- renku.core.workflow.plan_factory.delete_indirect_files_list(working_dir)[source]¶
Remove indirect inputs, outputs, and parameters list.
- renku.core.workflow.plan_factory.get_indirect_inputs_path(project_path)[source]¶
Return path to file that contains indirect inputs list.
- renku.core.workflow.plan_factory.get_indirect_outputs_path(project_path)[source]¶
Return path to file that contains indirect outputs list.
- renku.core.workflow.plan_factory.get_indirect_parameters_path(project_path)[source]¶
Return path to file that contains indirect parameters list.
- renku.core.workflow.plan_factory.read_files_list(files_list)[source]¶
Read files list yaml containing name:path pairs.
- renku.core.workflow.plan_factory.read_indirect_parameters(working_dir)[source]¶
Read and return indirect parameters.
Represent the Common Workflow Language types.
Resolution of Workflow
execution values precedence.
- class renku.core.workflow.value_resolution.CompositePlanValueResolver(plan, values=None)[source]¶
Bases:
renku.core.workflow.value_resolution.ValueResolver
Value resolution class for a
CompositePlan
.Applies values and default_values to a nested workflow.
Order of precedence is as follows (from lowest to highest): - Default value on a parameter - Default value on a mapping to the parameter - Value passed to a mapping to the parameter - Value passed to the parameter - Value propagated to a parameter from the source of a ParameterLink
- class renku.core.workflow.value_resolution.PlanValueResolver(plan, values)[source]¶
Bases:
renku.core.workflow.value_resolution.ValueResolver
Value resolution class for a
Plan
.Applies values and default_values to a workflow.
- class renku.core.workflow.value_resolution.ValueResolver(plan, values)[source]¶
Bases:
abc.ABC
Value resolution class for an
AbstractPlan
.- abstract apply()[source]¶
Applies values and default_values to a potentially nested workflow.
- Returns
The
AbstractPlan
with the user provided values set.- Return type
- static get(plan, values)[source]¶
Factory method to obtain the specific ValueResolver for a workflow.
- Parameters
plan (AbstractPlan) – a workflow.
values (Dict[str, Any]) – user defined dictionary of runtime values for the provided workflow.
- Returns
A ValueResolver object.
- Return type
“ValueResolver”
- renku.core.workflow.value_resolution.apply_parameter_links(workflow)[source]¶
Apply values from parameter links.
- Parameters
workflow (CompositePlan) – The workflow whose links values should be applied on.
Workflow file core logic.
- renku.core.workflow.workflow_file.filter_steps(workflow, steps)[source]¶
Return a subset of workflow file steps.
Sessions¶
Docker based interactive session provider.
- class renku.core.session.docker.DockerSessionProvider[source]¶
Bases:
renku.domain_model.session.ISessionProvider
A docker based interactive session provider.
- docker_client()[source]¶
Get the docker client.
Note
This is not a @property, even though it should be, because
pluggy
will call it in that case in unrelated parts of the code that will- Raises
errors.DockerError – Exception when docker is not available.
- Returns
The docker client.
- property name¶
Return session provider’s name.
- session_list(project_name, config)[source]¶
Lists all the sessions currently running by the given session provider.
- Returns
a list of sessions.
- Return type
list
- session_open(project_name, session_name, **kwargs)[source]¶
Open a given interactive session.
- Parameters
project_name (str) – Renku project name.
session_name (str) – The unique id of the interactive session.
- session_start(image_name, project_name, config, cpu_request=None, mem_request=None, disk_request=None, gpu_request=None, **kwargs)[source]¶
Creates an interactive session.
- Returns
Provider message and a possible warning message.
- Return type
Tuple[str, str]
Interactive session business logic.
- class renku.core.session.session.SessionList(sessions, all_local, warning_messages)¶
Bases:
tuple
Create new instance of SessionList(sessions, all_local, warning_messages)
- all_local¶
Alias for field number 1
- sessions¶
Alias for field number 0
- warning_messages¶
Alias for field number 2
- renku.core.session.session.session_list(config_path, provider=None)[source]¶
List interactive sessions.
- Parameters
config_path (str, optional) – Path to config YAML.
provider (str, optional) – Name of the session provider to use.
- Returns
The list of sessions, whether they’re all local sessions and potential warnings raised.
- renku.core.session.session.session_open(session_name, provider=None, **kwargs)[source]¶
Open interactive session in the browser.
- Parameters
session_name (str) – Name of the session to open.
provider (str, optional) – Name of the session provider to use.
- renku.core.session.session.session_start(config_path, provider, image_name=None, cpu_request=None, mem_request=None, disk_request=None, gpu_request=None, **kwargs)[source]¶
Start interactive session.
- Parameters
config_path (str, optional) – Path to config YAML.
provider (str, optional) – Name of the session provider to use.
image_name (str, optional) – Image to start.
cpu_request (float, optional) – Number of CPUs to request.
mem_request (str, optional) – Size of memory to request.
disk_request (str, optional) – Size of disk to request (if supported by provider).
gpu_request (str, optional) – Number of GPUs to request.
- renku.core.session.session.session_stop(session_name, stop_all=False, provider=None)[source]¶
Stop interactive session.
- Parameters
session_name (str) – Name of the session to open.
stop_all (bool) – Whether to stop all sessions or just the specified one.
provider (str, optional) – Name of the session provider to use.
Templates¶
Template management.
- class renku.core.template.template.EmbeddedTemplates(path, source, reference, version, skip_validation=False)[source]¶
Bases:
renku.domain_model.template.TemplatesSource
Represent templates that are bundled with Renku.
For embedded templates,
source
is “renku”. In the old versioning scheme,version
is set to the installed Renku version andreference
is not set. In the new scheme, bothversion
andreference
are set to the template version.
- class renku.core.template.template.FileAction(value)[source]¶
Bases:
enum.IntEnum
Types of operation when copying a template to a project.
- class renku.core.template.template.RepositoryTemplates(path, source, reference, version, repository, skip_validation=False)[source]¶
Bases:
renku.domain_model.template.TemplatesSource
Represent a local/remote template repository.
A template repository is checked out at a specific Git reference if one is provided. However, it’s still possible to get available versions of templates.
For these templates,
reference
is set to whatever user passed as a reference (defaults to remote HEAD if not passed) andversion
is set to the commit SHA of the reference commit.- get_all_references(id)[source]¶
Return a list of git tags that are valid SemVer and include a template id.
- class renku.core.template.template.TemplateAction(value)[source]¶
Bases:
enum.Enum
Types of template rendering.
- renku.core.template.template.copy_template_to_project(rendered_template, project, actions, cleanup=True)[source]¶
Update project files and metadata from a template.
- renku.core.template.template.get_file_actions(rendered_template, template_action, interactive)[source]¶
Render a template regarding files in a project.
- renku.core.template.template.has_template_checksum()[source]¶
Return if project has a templates checksum file.
- renku.core.template.template.is_renku_template(source)[source]¶
Return if template comes from Renku.
- renku.core.template.template.read_template_checksum()[source]¶
Read templates checksum file for a project.
- renku.core.template.template.set_template_parameters(template, template_metadata, input_parameters, interactive=False)[source]¶
Set and verify template parameters’ values in the template_metadata.
- renku.core.template.template.write_template_checksum(checksums)[source]¶
Write templates checksum file for a project.
Template use cases.
- renku.core.template.usecase.check_for_template_update(project)[source]¶
Check if the project can be updated to a newer version of the project template.
- renku.core.template.usecase.does_dockerfile_contain_only_version_change()[source]¶
Return True if Dockerfile only contains Renku version changes.
- renku.core.template.usecase.is_dockerfile_updated_by_user()[source]¶
Return if user modified the
Dockerfile
.
- renku.core.template.usecase.list_templates(source, reference)[source]¶
Return available templates from a source.
- renku.core.template.usecase.select_template(templates_source, id=None)[source]¶
Select a template from a template source.
- renku.core.template.usecase.set_template(source, reference, id, force, interactive, input_parameters, dry_run)[source]¶
Set template for a project.
- renku.core.template.usecase.update_dockerfile_checksum(new_checksum)[source]¶
Update
Dockerfile
template checksum if possible.
- renku.core.template.usecase.update_template(force, interactive, dry_run)[source]¶
Update project’s template if possible. Return True if updated.
- renku.core.template.usecase.validate_templates(source=None, reference=None)[source]¶
Validate a template repository.
- Parameters
source (str, optional) – Remote repository URL to clone and check (Default value = None).
reference (str, optional) – Git commit/branch/tag to check (Default value = None).
- Returns
- Dictionary containing errors and warnings for manifest and
templates, along with a
valid
field telling if all checks passed.
- Return type
Dict[str, Union[str, Dict[str, List[str]]]]
Errors¶
Errors that can be raised by renku.core
.
Renku exceptions.
- exception renku.core.errors.ActivityDownstreamNotEmptyError(activity)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when an activity cannot be deleted because its downstream is not empty.
- exception renku.core.errors.AuthenticationError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when there is a problem with authentication.
- exception renku.core.errors.ChildWorkflowNotFoundError(child, workflow)[source]¶
Bases:
renku.core.errors.WorkflowError
Raised when a child could not be found on a composite workflow.
Embed exception and build a custom message.
- exception renku.core.errors.CommandFinalizedError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when trying to modify a finalized command builder.
- exception renku.core.errors.CommandNotFinalizedError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a non-finalized command is executed.
- exception renku.core.errors.CommitMessageEmpty[source]¶
Bases:
renku.core.errors.RenkuException
Raise invalid commit message.
Build a custom message.
- exception renku.core.errors.CommitProcessingError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a commit couldn’t be processed during graph build.
- exception renku.core.errors.ConfigurationError[source]¶
Bases:
renku.core.errors.RenkuException
Raise in case of misconfiguration; use GitConfigurationError for git-related configuration errors.
- exception renku.core.errors.DatasetException[source]¶
Bases:
renku.core.errors.RenkuException
Base class for all dataset-related exceptions.
- exception renku.core.errors.DatasetExistsError(name)[source]¶
Bases:
renku.core.errors.DatasetException
Raise when trying to create an existing dataset.
- exception renku.core.errors.DatasetImageError[source]¶
Bases:
renku.core.errors.DatasetException
Raised when a local dataset image is not accessible.
- exception renku.core.errors.DatasetImportError[source]¶
Bases:
renku.core.errors.DatasetException
Raised when a dataset cannot be imported/pulled from a remote source.
- exception renku.core.errors.DatasetNotFound(*, name=None, message=None)[source]¶
Bases:
renku.core.errors.DatasetException
Raise when dataset is not found.
Build a custom message.
- exception renku.core.errors.DatasetProviderNotFound(*, name=None, uri=None, message=None)[source]¶
Bases:
renku.core.errors.DatasetException
,renku.core.errors.ParameterError
Raised when a dataset provider cannot be found based on a URI or a provider name.
- exception renku.core.errors.DatasetTagNotFound(tag)[source]¶
Bases:
renku.core.errors.DatasetException
Raise when a tag can’t be found.
- exception renku.core.errors.DirectoryNotEmptyError(path)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a directory passed as output is not empty.
Build a custom message.
- exception renku.core.errors.DirtyRenkuDirectory(repository)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a directory in the renku repository is dirty.
Build a custom message.
- exception renku.core.errors.DirtyRepository(repository)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when trying to work with dirty repository.
Build a custom message.
- exception renku.core.errors.DockerAPIError(reason)[source]¶
Bases:
renku.core.errors.DockerError
Raised when error has returned from the Docker API.
Embed exception and build a custom message.
- exception renku.core.errors.DockerError(reason)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when error has occurred while executing docker command.
Embed exception and build a custom message.
- exception renku.core.errors.DockerfileUpdateError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when the renku version in the Dockerfile couldn’t be updated.
- exception renku.core.errors.DuplicateWorkflowNameError[source]¶
Bases:
renku.core.errors.WorkflowError
Raises when a workflow name already exists.
- exception renku.core.errors.ExportError[source]¶
Bases:
renku.core.errors.DatasetException
Raised when a dataset cannot be exported.
- exception renku.core.errors.ExternalFileNotFound(path)[source]¶
Bases:
renku.core.errors.DatasetException
Raise when an external file is not found.
Build a custom message.
- exception renku.core.errors.ExternalStorageDisabled[source]¶
Bases:
renku.core.errors.RenkuException
Raise when disabled repository storage API is trying to be used.
Build a custom message.
- exception renku.core.errors.ExternalStorageNotInstalled[source]¶
Bases:
renku.core.errors.RenkuException
Raise when LFS is required but not found or installed in the repository.
Build a custom message.
- exception renku.core.errors.FailedMerge(repository, branch, merge_args)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when automatic merge failed.
Build a custom message.
- exception renku.core.errors.FileNotFound(path, checksum=None, revision=None)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a file is not found.
Build a custom message.
- exception renku.core.errors.GitCommandError(message='Git command failed.', command=None, stdout=None, stderr=None, status=None)[source]¶
Bases:
renku.core.errors.GitError
Raised when a Git command fails.
Build a custom message.
- exception renku.core.errors.GitCommitNotFoundError[source]¶
Bases:
renku.core.errors.GitError
Raised when a commit cannot be found in a Repository.
- exception renku.core.errors.GitConfigurationError[source]¶
Bases:
renku.core.errors.GitError
Raised when a git configuration cannot be accessed.
- exception renku.core.errors.GitError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a Git operation fails.
- exception renku.core.errors.GitLFSError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a Git LFS operation fails.
- exception renku.core.errors.GitMissingEmail(message=None)[source]¶
Bases:
renku.core.errors.GitConfigurationError
Raise when the email is not configured.
Build a custom message.
- exception renku.core.errors.GitMissingUsername(message=None)[source]¶
Bases:
renku.core.errors.GitConfigurationError
Raise when the username is not configured.
Build a custom message.
- exception renku.core.errors.GitReferenceNotFoundError[source]¶
Bases:
renku.core.errors.GitError
Raised when a branch or a reference cannot be found.
- exception renku.core.errors.GitRemoteNotFoundError[source]¶
Bases:
renku.core.errors.GitError
Raised when a remote cannot be found.
- exception renku.core.errors.GraphCycleError(cycles, message=None)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a parameter reference cannot be resolved to a parameter.
Embed exception and build a custom message.
- exception renku.core.errors.IncompatibleParametersError(first_param=None, second_param=None)[source]¶
Bases:
renku.core.errors.ParameterError
Raise in case of incompatible parameters/flags.
Build a custom message.
- exception renku.core.errors.InvalidAccessToken[source]¶
Bases:
renku.core.errors.RenkuException
Raise when access token is incorrect.
Build a custom message.
- exception renku.core.errors.InvalidFileOperation[source]¶
Bases:
renku.core.errors.RenkuException
Raise when trying to perform invalid file operation.
- exception renku.core.errors.InvalidGitURL[source]¶
Bases:
renku.core.errors.GitError
Raise when a Git URL is not valid.
- exception renku.core.errors.InvalidInputPath[source]¶
Bases:
renku.core.errors.RenkuException
Raise when input path does not exist or is not in the repository.
- exception renku.core.errors.InvalidOutputPath[source]¶
Bases:
renku.core.errors.RenkuException
Raise when trying to work with an invalid output path.
- exception renku.core.errors.InvalidSuccessCode(return_code, success_codes=None, message=None)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when the exit-code is not 0 or redefined.
Build a custom message.
- exception renku.core.errors.InvalidTemplateError[source]¶
Bases:
renku.core.errors.TemplateError
Raised when using a non-valid template.
- exception renku.core.errors.KeyNotFoundError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when an SSH private or public key couldn’t be found.
- exception renku.core.errors.LockError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a project cannot be locked.
- exception renku.core.errors.MappingExistsError(existing_mappings)[source]¶
Bases:
renku.core.errors.WorkflowError
Raised when a parameter mapping exists already.
Embed exception and build a custom message.
- exception renku.core.errors.MappingNotFoundError(mapping, workflow)[source]¶
Bases:
renku.core.errors.WorkflowError
Raised when a parameter mapping does not exist.
Embed exception and build a custom message.
- exception renku.core.errors.MetadataMergeError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when merging of metadata failed.
- exception renku.core.errors.MigrationError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when something went wrong during migrations.
- exception renku.core.errors.MigrationRequired[source]¶
Bases:
renku.core.errors.RenkuException
Raise when migration is required.
Build a custom message.
- exception renku.core.errors.MinimumVersionError(current_version, minimum_version)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when accessing a project whose minimum version is larger than the current renku version.
- exception renku.core.errors.NodeNotFoundError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when NodeJs is not installed on the system.
Build a custom message.
- exception renku.core.errors.NotFound[source]¶
Bases:
renku.core.errors.RenkuException
Raise when an object is not found in KG.
- exception renku.core.errors.NotebookSessionImageNotExistError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a user attempts to start a session with an image that does not exist.
- exception renku.core.errors.NotebookSessionNotReadyError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a user attempts to open a session that is not ready.
- exception renku.core.errors.NothingToCommit[source]¶
Bases:
renku.core.errors.RenkuException
Raise when there is nothing to commit.
Build a custom message.
- exception renku.core.errors.NothingToExecuteError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a rerun/update command does not execute any workflows.
- exception renku.core.errors.ObjectNotFoundError(filename)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when an object is not found in the storage.
Embed exception and build a custom message.
- exception renku.core.errors.OperationError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when an operation at runtime raises an error.
- exception renku.core.errors.OutputsNotFound[source]¶
Bases:
renku.core.errors.RenkuException
Raise when there are not any detected outputs in the repository.
Build a custom message.
- exception renku.core.errors.ParameterError(message, param_hint=None, show_prefix=True)[source]¶
Bases:
renku.core.errors.RenkuException
Raise in case of invalid parameter.
Build a custom message.
- exception renku.core.errors.ParameterLinkError(reason)[source]¶
Bases:
renku.core.errors.RenkuException
Raised when a parameter link cannot be created.
Embed exception and build a custom message.
- exception renku.core.errors.ParameterNotFoundError(parameter, workflow)[source]¶
Bases:
renku.core.errors.WorkflowError
Raised when a parameter reference cannot be resolved to a parameter.
Embed exception and build a custom message.
- exception renku.core.errors.ParseError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a workflow file command has invalid format.
- exception renku.core.errors.ProjectContextError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when no project context is pushed or there is a project context-related error.
- exception renku.core.errors.ProjectNotFound[source]¶
Bases:
renku.core.errors.RenkuException
Raise when one or more projects couldn’t be found in the KG.
- exception renku.core.errors.ProjectNotSupported[source]¶
Bases:
renku.core.errors.RenkuException
Raise when project version is newer than the supported version.
Build a custom message.
- exception renku.core.errors.ProtectedFiles(ignored)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when trying to work with protected files.
Build a custom message.
- exception renku.core.errors.RCloneException[source]¶
Bases:
renku.core.errors.DatasetException
Base class for all rclone-related exceptions.
- exception renku.core.errors.RenkuException[source]¶
Bases:
Exception
A base class for all Renku related exception.
You can catch all errors raised by Renku SDK by using
except RenkuException:
.
- exception renku.core.errors.RenkuSaveError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when renku save doesn’t work.
- exception renku.core.errors.RenkulabSessionError[source]¶
Bases:
renku.core.errors.SessionStartError
Raised when an error occurs trying to start sessions with the notebook service.
- exception renku.core.errors.RenkulabSessionGetUrlError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when Renku deployment’s URL cannot be gotten from project’s remotes or configured remotes.
- exception renku.core.errors.RequestError[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a
requests
call fails.
- exception renku.core.errors.SHACLValidationError[source]¶
Bases:
renku.core.errors.RenkuException
Raises when SHACL validation of the graph fails.
- exception renku.core.errors.SSHNotFoundError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when SSH client is not installed on the system.
Build a custom message.
- exception renku.core.errors.SSHNotSetupError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when SSH client is not installed on the system.
Build a custom message.
- exception renku.core.errors.SessionStartError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when an error occurs trying to start sessions.
- exception renku.core.errors.StorageObjectNotFound(error=None)[source]¶
Bases:
renku.core.errors.RCloneException
Raised when a file or directory cannot be found in the remote storage.
- exception renku.core.errors.StorageProviderNotFound(uri)[source]¶
Bases:
renku.core.errors.DatasetException
,renku.core.errors.ParameterError
Raised when a storage provider cannot be found based on a URI.
- exception renku.core.errors.TemplateError[source]¶
Bases:
renku.core.errors.RenkuException
Base class for template-related exceptions.
- exception renku.core.errors.TemplateMissingReferenceError[source]¶
Bases:
renku.core.errors.TemplateError
Raised when using a non-valid template.
- exception renku.core.errors.TemplateNotFoundError[source]¶
Bases:
renku.core.errors.TemplateError
Raised when a template cannot be found in a template source or at a specific reference.
- exception renku.core.errors.TemplateUpdateError[source]¶
Bases:
renku.core.errors.TemplateError
Raised when a project couldn’t be updated from its template.
- exception renku.core.errors.TerminalSizeError[source]¶
Bases:
renku.core.errors.RenkuException
Raised when terminal is too small for a command.
- exception renku.core.errors.UninitializedProject(repo_path)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when a project does not seem to have been initialized yet.
Build a custom message.
- exception renku.core.errors.UnmodifiedOutputs(repository, unmodified)[source]¶
Bases:
renku.core.errors.RenkuException
Raise when there are unmodified outputs in the repository.
Build a custom message.
- exception renku.core.errors.UsageError[source]¶
Bases:
renku.core.errors.RenkuException
Raise in case of unintended usage of certain function calls.
- exception renku.core.errors.WorkflowError[source]¶
Bases:
renku.core.errors.RenkuException
Base class for workflow-related errors.
- exception renku.core.errors.WorkflowExecuteError(fail_reason=None, show_prefix=True)[source]¶
Bases:
renku.core.errors.WorkflowError
Raises when a workflow execution fails.
Build a custom message.
- exception renku.core.errors.WorkflowExportError[source]¶
Bases:
renku.core.errors.WorkflowError
Raises when a workflow cannot be exported.
- exception renku.core.errors.WorkflowNotFoundError(name_or_id)[source]¶
Bases:
renku.core.errors.WorkflowError
Raised when a workflow could not be found.
Embed exception and build a custom message.
- exception renku.core.errors.WorkflowRerunError(cwl_file)[source]¶
Bases:
renku.core.errors.WorkflowError
Raises when a workflow re-execution fails.
Build a custom message.
Utilities¶
Communicator classes for printing output.
- class renku.core.util.communication.CommunicationCallback[source]¶
Bases:
object
Base communication callback class.
- renku.core.util.communication.confirm(msg, abort=False, warning=False, default=False)[source]¶
Get confirmation for an action from all listeners.
- renku.core.util.communication.disable()[source]¶
Disable all outputs; by default everything is enabled.
- renku.core.util.communication.finalize_progress(name)[source]¶
End a progress tracker on all listeners.
- renku.core.util.communication.has_prompt()[source]¶
Return True if communicator provides a direct prompt to users.
- renku.core.util.communication.prompt(msg, type=None, default=None, **kwargs)[source]¶
Show a message prompt.
- renku.core.util.communication.start_progress(name, total, **kwargs)[source]¶
Start a progress tracker on all listeners.
- renku.core.util.communication.update_progress(name, amount)[source]¶
Update a progress tracker on all listeners.
Implement various context managers.
- class renku.core.util.contexts.Isolation(**kwargs)[source]¶
Bases:
contextlib.ExitStack
Isolate execution.
Create a context manager.
- renku.core.util.contexts.Lock(filename, timeout=0, mode='shared', blocking=False)[source]¶
A file-based lock context manager.
- renku.core.util.contexts.measure(message='TOTAL')[source]¶
Measure execution time of enclosing code block.
- class renku.core.util.contexts.redirect_stdin(new_target)[source]¶
Bases:
contextlib.ContextDecorator
Implement missing redirect stdin based on
contextlib.py
.Keep the original stream.
- renku.core.util.contexts.renku_project_context(path, check_git_path=True)[source]¶
Provide a project context with repo path injected.
- renku.core.util.contexts.wait_for(delay)[source]¶
Make sure that at least
delay
seconds are passed during the execution of the wrapped code block.
- renku.core.util.contexts.with_project_metadata(project_gateway, database_gateway, read_only=False, name=None, namespace=None, description=None, keywords=None, custom_metadata=None)[source]¶
Yield an editable metadata object.
- Parameters
project_gateway (IProjectGateway) – Injected project gateway.
database_gateway (IDatabaseGateway) – Injected database gateway.
read_only (bool) – Whether to save changes or not (Default value = False).
name (Optional[str]) – Name of the project (when creating a new one) (Default value = None).
namespace (Optional[str]) – Namespace of the project (when creating a new one) (Default value = None).
description (Optional[str]) – Project description (when creating a new one) (Default value = None).
keywords (Optional[List[str]]) – Keywords for the project (when creating a new one) (Default value = None).
custom_metadata (Optional[Dict]) – Custom JSON-LD metadata (when creating a new project) (Default value = None).
Renku datetime utilities.
- renku.core.util.datetime8601.fix_datetime(value)[source]¶
Fix timezone of non-aware datetime objects and remove microseconds.
- renku.core.util.datetime8601.local_now(remove_microseconds=False)[source]¶
Return current datetime in local timezone.
- renku.core.util.datetime8601.match_iso8601(string, pos=0, endpos=9223372036854775807)¶
Matches zero or more characters at the beginning of the string.
- renku.core.util.datetime8601.validate_iso8601(str_val)[source]¶
Check if datetime string is in ISO8601 format.
Helper utilities for handling DOIs.
Helper utilities for handling file size strings.
- renku.core.util.file_size.bytes_to_unit(size_in_bytes, unit)[source]¶
Return size in the provided unit.
- renku.core.util.file_size.parse_file_size(size_str)[source]¶
Parse a human readable file size to bytes.
Git utility functions.
- renku.core.util.git.check_global_git_user_is_configured()[source]¶
Check that git user information is configured.
- renku.core.util.git.clone_renku_repository(url, path, gitlab_token=None, deployment_hostname=None, depth=None, install_githooks=False, install_lfs=True, skip_smudge=True, recursive=True, progress=None, config=None, raise_git_except=False, checkout_revision=None, use_renku_credentials=False, reuse_existing_repository=False)[source]¶
Clone a Renku Repository.
- Parameters
url (str) – The Git URL to clone.
path (Union[Path, str]) – The path to clone into.
gitlab_token – The gitlab OAuth2 token (Default value = None).
deployment_hostname – The hostname of the current renku deployment (Default value = None).
depth (Optional[int], optional) – The clone depth, number of commits from HEAD (Default value = None).
install_githooks – Whether to install git hooks (Default value = False).
install_lfs – Whether to install Git LFS (Default value = True).
skip_smudge – Whether to pull files from Git LFS (Default value = True).
recursive – Whether to clone recursively (Default value = True).
progress – The GitProgress object (Default value = None).
config (Optional[dict], optional) – Set configuration for the project (Default value = None).
raise_git_except – Whether to raise git exceptions (Default value = False).
checkout_revision – The revision to checkout after clone (Default value = None).
use_renku_credentials (bool, optional) – Whether to use Renku provided credentials (Default value = False).
reuse_existing_repository (bool, optional) – Whether to clone over an existing repository (Default value = False).
- Returns
The cloned repository.
- renku.core.util.git.clone_repository(url, path=None, install_githooks=True, install_lfs=True, skip_smudge=True, recursive=True, depth=None, progress=None, config=None, raise_git_except=False, checkout_revision=None, no_checkout=False, clean=False, clone_options=None)[source]¶
Clone a Git repository and install Git hooks and LFS.
- Parameters
url – The Git URL to clone.
path (Union[Path, str], optional) – The path to clone into (Default value = None).
install_githooks – Whether to install git hooks (Default value = True).
install_lfs – Whether to install Git LFS (Default value = True).
skip_smudge – Whether to pull files from Git LFS (Default value = True).
recursive – Whether to clone recursively (Default value = True).
depth – The clone depth, number of commits from HEAD (Default value = None).
progress – The GitProgress object (Default value = None).
config (Optional[dict], optional) – Set configuration for the project (Default value = None).
raise_git_except – Whether to raise git exceptions (Default value = False).
checkout_revision – The revision to checkout after clone (Default value = None).
no_checkout (bool, optional) – Whether to perform a checkout (Default value = False).
clean (bool, optional) – Whether to require the target folder to be clean (Default value = False).
clone_options (List[str], optional) – Additional clone options (Default value = None).
- Returns
The cloned repository.
- renku.core.util.git.commit_changes(*paths, repository, message=None)[source]¶
Commit paths to the repository.
- Parameters
*paths (Union[Path, str]) – The paths to commit.
repository (Repository) – The repository to commit to.
message – The commit message (Default value = None).
- Raises
errors.GitError – If paths couldn’t be committed.
- Returns
List of paths that were committed.
- renku.core.util.git.create_backup_remote(repository, remote_name, url)[source]¶
Create a backup for
remote_name
and sets its url tourl
.- Parameters
repository (Repository) – The current repository.
remote_name (str) – The name of the backup remote.
url (str) – The remote URL.
- Returns
- Tuple of backup remote name, whether it existed already and the created
remote if successful.
- Return type
Tuple[str, bool, Optional[Remote]]
- renku.core.util.git.finalize_commit(*, diff_before, repository, transaction_id, commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True, skip_staging=False)[source]¶
Commit modified/added paths.
- renku.core.util.git.get_cache_directory_for_repository(url)[source]¶
Return a path to project’s cache directory.
- Parameters
url – The repository URL.
- Returns
The path of the cache.
- Return type
Path
- renku.core.util.git.get_committer_agent(commit)[source]¶
Return committer SoftwareAgent.
- Parameters
commit (Commit) – The commit to check.
- Returns
The agent responsible for the commit.
- Return type
- renku.core.util.git.get_dirty_paths(repository)[source]¶
Get paths of dirty files in the repository.
- renku.core.util.git.get_entity_from_revision(repository, path, revision=None, bypass_cache=False, checksum=None)[source]¶
Return an Entity instance from given path and revision.
- Parameters
repository (Repository) – The current repository.
path (Union[Path, str]) – The path of the entity.
revision (str, optional) – The revision to check at (Default value = None).
bypass_cache (bool) – Whether to ignore cached entries and get information from disk (Default value = False).
checksum (str, optional) – Pre-calculated checksum for performance reasons, will be calculated if not set.
- Returns
The Entity for the given path and revision.
- Return type
- renku.core.util.git.get_file_size(repository_path, path)[source]¶
Return file size for a file inside a git repository.
- renku.core.util.git.get_full_repository_path(url)[source]¶
Extract hostname/path of a git repository from its URL.
- Parameters
url (Optional[str]) – The URL.
- Returns
The hostname plus path extracted from the URL.
- renku.core.util.git.get_git_repository(path='.')[source]¶
Get Git repository from the current path or any of its parents.
- Parameters
path – Path to start from (Default value = “.”).
- Raises
ValueError – If not inside a git repository.
- Returns
Git repository
- renku.core.util.git.get_git_user(repository)[source]¶
Return git user.
- Parameters
repository (Optional[Repository]) – The Git repository.
- Returns
The person associated with the repository.
- Return type
Optional[Person]
- renku.core.util.git.get_hook_path(path, name)[source]¶
Return path to the given named hook in the given repository.
- Parameters
path (Path) – The current Git repository’s path.
name (str) – The name of the hook.
- Returns
Path to the hook.
- Return type
Path
- renku.core.util.git.get_in_submodules(repository, commit, path)[source]¶
Resolve filename in submodules.
- renku.core.util.git.get_oauth_url(url, gitlab_token)[source]¶
Format URL with a username and password.
- Parameters
url – The URL to format.
gitlab_token – The Gitlab OAuth2 Token.
- Returns
The URL with credentials added.
- renku.core.util.git.get_remote(repository, *, name=None, url=None)[source]¶
Return repository’s remote using its name or url or return default remote if any.
- Parameters
repository (Optional[Repository]) – The Git repository.
name (str, optional) – The name of the remote (Default value = None).
url (str, optional) – The remote URL (Default value = None).
- Returns
The remote, if found.
- Return type
Optional[Remote]
- renku.core.util.git.get_renku_repo_url(remote_url, deployment_hostname=None, access_token=None)[source]¶
Return a repo url that can be authenticated by renku.
- Parameters
remote_url – The repository URL.
deployment_hostname – The host name used by this deployment (Default value = None).
access_token – The OAuth2 access token (Default value = None).
- Returns
The Renku repository URL with credentials.
- renku.core.util.git.get_repository_name(url)[source]¶
Extract name of a git repository from its URL.
- Parameters
url (str) – The URL to get the repository name from.
- Returns
The repository name.
- Return type
str
- renku.core.util.git.have_same_remote(url1, url2)[source]¶
Checks if two git urls point to the same remote repo ignoring protocol and credentials.
- Parameters
url1 – The first URL.
url2 – The second URL.
- Returns
True if both URLs point to the same repository.
- Return type
bool
- renku.core.util.git.is_path_safe(path)[source]¶
Check if the path should be used in output.
- Parameters
path (Union[Path, str]) – The path to check.
- Returns
True if the path is safe else False.
- Return type
bool
- renku.core.util.git.is_valid_git_repository(repository)[source]¶
Return if is a git repository and has a valid HEAD.
- Parameters
repository (Optional[Repository]) – The repository to check.
- Returns
Whether or not this is a valid Git repository.
- Return type
bool
- renku.core.util.git.parse_git_url(url)[source]¶
Return parsed git url.
- Parameters
url (Optional[str]) – The URL to parse.
- Raises
errors.InvalidGitURL – If
url
is empty.- Returns
The parsed GitURL.
- Return type
- renku.core.util.git.prepare_commit(*, repository, commit_only=None, skip_dirty_checks=False, skip_staging=False)[source]¶
Gather information about repo needed for committing later on.
- renku.core.util.git.push_changes(repository, remote=None, reset=True)[source]¶
Push to a remote branch. If the remote branch is protected a new remote branch will be created and pushed to.
- Parameters
repository (Repository) – The current repository.
remote (str, optional) – The remote to push to (Default value = None).
reset (bool, optional) – Whether to reset active branch to its upstream branch, used if changes get pushed to a temporary branch (Default value = True).
- Raises
errors.GitError – If there’s no remote or the push fails.
- Returns
Name of the branch that was pushed to.
- Return type
str
- renku.core.util.git.run_command(command, *paths, separator=None, **kwargs)[source]¶
Execute command by splitting paths to make sure that argument list will be within os limits.
- Parameters
command – A list or tuple containing command and its arguments.
*paths – Paths to run on.
separator – Separator for paths if they need to be passed as string. (Default value = None)
- Raises
errors.GitError – If a Git subcommand failed.
- Returns
Result of last invocation.
- renku.core.util.git.set_git_credential_helper(repository, hostname)[source]¶
Set up credential helper for renku git.
- renku.core.util.git.shorten_message(message, line_length=100, body_length=65000)[source]¶
Wraps and shortens a commit message.
- Parameters
message (str) – message to adjust.
line_length (int, optional) – maximum line length before wrapping. 0 for infinite (Default value = 100).
body_length (int, optional) – maximum body length before cut. 0 for infinite (Default value = 65000).
- Raises
ParameterError – If line_length or body_length < 0
- Returns
message wrapped and trimmed.
- renku.core.util.git.with_commit(*, repository, transaction_id, commit_only=None, commit_empty=True, raise_if_empty=False, commit_message=None, abbreviate_message=True, skip_dirty_checks=False)[source]¶
Automatic commit.
Helpers functions for metadata management/parsing.
- renku.core.util.metadata.construct_creator(creator, ignore_email)[source]¶
Parse input and return an instance of Person.
- renku.core.util.metadata.construct_creators(creators, ignore_email=False)[source]¶
Parse input and return a list of Person.
- renku.core.util.metadata.is_external_file(path, project_path)[source]¶
Checks if a path is an external file.
- renku.core.util.metadata.make_project_temp_dir(project_path)[source]¶
Create a temporary directory inside project’s temp path.
- renku.core.util.metadata.prompt_for_credentials(provider_credentials)[source]¶
Prompt for provider credentials if needed and update and store them.
- renku.core.util.metadata.read_renku_version_from_dockerfile(path=None)[source]¶
Read RENKU_VERSION from the content of path if a valid version is available.
- renku.core.util.metadata.replace_renku_version_in_dockerfile(dockerfile_content, version)[source]¶
Replace Renku version in the Dockerfile.
- renku.core.util.metadata.store_credentials(section, key, value)[source]¶
Write provider’s credentials.
OS utility functions.
Return True if paths are equal or one is the parent of the other.
- renku.core.util.os.create_symlink(path, symlink_path, overwrite=True)[source]¶
Create a symlink that points from symlink_path to path.
- renku.core.util.os.delete_dataset_file(filepath, ignore_errors=True, follow_symlinks=False)[source]¶
Remove a file/symlink and its pointer file (for external files).
- renku.core.util.os.get_absolute_path(path, base=None, resolve_symlinks=False)[source]¶
Return absolute normalized path.
- renku.core.util.os.get_relative_path(path, base, strict=False)[source]¶
Return a relative path to the base if path is within base with/without resolving symlinks.
- renku.core.util.os.get_relative_path_to_cwd(path)[source]¶
Get a relative path to current working directory.
- renku.core.util.os.get_relative_paths(paths, base)[source]¶
Return a list of paths relative to a base path.
- renku.core.util.os.get_safe_relative_path(path, base)[source]¶
Return a relative path to the base and check path is within base with all symlinks resolved.
NOTE: This is used to prevent path traversal attack.
- renku.core.util.os.hash_file(path, hash_type='sha256')[source]¶
Calculate the sha256 hash of a file.
- renku.core.util.os.hash_file_descriptor(file, hash_type='sha256')[source]¶
Hash content of a file descriptor.
- renku.core.util.os.normalize_to_ascii(input_string, sep='-')[source]¶
Convert a string to only contain ASCII characters, with non-ASCII substring replaced with
sep
.
- renku.core.util.os.safe_read_yaml(path)[source]¶
Parse a YAML file.
- Returns
In case of success a dictionary of the YAML’s content, otherwise raises a ParameterError exception.
Utility for working with HTTP session.
This module provides some wrapper functions around requests
library. It sets a timeout and converts exception types
whenever needed. Use this module instead of requests
.
- renku.core.util.requests.download_file(base_directory, url, filename, extract, chunk_size=16384)[source]¶
Download a URL to a given location.
- renku.core.util.requests.get_filename_from_headers(response)[source]¶
Extract filename from content-disposition headers if available.
- renku.core.util.requests.get_redirect_url(url)[source]¶
Return redirect URL if any; otherwise, return the original URL.
- renku.core.util.requests.head(url, *, allow_redirects=False, headers=None)[source]¶
Send a HEAD request.
- renku.core.util.requests.post(url, *, data=None, files=None, headers=None, json=None, params=None)[source]¶
Send a POST request.
- renku.core.util.requests.put(url, *, data=None, files=None, headers=None, params=None)[source]¶
Send a PUT request.
JSON-LD SHACL validations.
- renku.core.util.shacl.validate_graph(graph, shacl_path=None, format='nquads')[source]¶
Validate the current graph with a SHACL schema.
Uses default schema if not supplied.
Helper utilities for handling URLs.
- renku.core.util.urls.get_host(use_project_context=True)[source]¶
Return the hostname for the resource URIs.
Default is localhost. If RENKU_DOMAIN is set, it overrides the host from remote.
- renku.core.util.urls.get_slug(name, invalid_chars=None, lowercase=True)[source]¶
Create a slug from name.
- renku.core.util.urls.is_uri_subfolder(uri, subfolder_uri)[source]¶
Check if one uri is a ‘subfolder’ of another.
- renku.core.util.urls.parse_authentication_endpoint(endpoint=None, use_remote=False)[source]¶
Return a parsed url.
If an endpoint is provided then use it, otherwise, look for a configured endpoint. If no configured endpoint exists then try to use project’s remote url.
General utility functions.
- renku.core.util.util.NO_VALUE = <object object>¶
Sentinel to represent a value not being set in cases where
None
is a valid value.
- renku.core.util.util.NoValueType¶
Type to represent a value not being set in cases where
None
is a valid value.alias of
object
- renku.core.util.util.to_string(value, strip=False)[source]¶
Return a string representation of value and return an empty string for None.
Helper utilities for handling UUIDs.
- renku.core.util.uuid.is_uuid(value)[source]¶
Check if value is UUID4.
Copied from https://stackoverflow.com/questions/19989481/
Support JSON-LD context in models.
- class renku.core.util.yaml.NoDatesSafeLoader(stream)[source]¶
Bases:
yaml.cyaml.CSafeLoader
Used to safely load basic python objects but ignore datetime strings.
- classmethod remove_implicit_resolver(tag_to_remove)[source]¶
Remove implicit resolvers for a particular tag.
Takes care not to modify resolvers in super classes.
We want to load datetimes as strings, not dates, because we go on to serialize as json which doesn’t have the advanced types of yaml, and leads to incompatibilities down the track.
Git Internals¶
Git repository management.
- renku.core.git.finalize_worktree(isolation, path, branch_name, delete, new_branch, merge_args=('--ff-only',), exception=None)[source]¶
Cleanup and merge a previously created Git worktree.
- renku.core.git.get_mapped_std_streams(lookup_paths, streams=('stdin', 'stdout', 'stderr'))[source]¶
Get a mapping of standard streams to given paths.
- renku.core.git.prepare_worktree(path=None, branch_name=None, commit=None)[source]¶
Set up a Git worktree to provide isolation.
- renku.core.git.with_worktree(path=None, branch_name=None, commit=None, merge_args=('--ff-only',))[source]¶
Create new worktree.
Git utilities.
- class renku.domain_model.git.GitURL(href, path=None, scheme='ssh', hostname='localhost', username=None, password=None, port=None, owner=None, name=None, slug=None, regex=None)[source]¶
Parser for common Git URLs.
Method generated by attrs for class GitURL.
- property image¶
Return image name.
- property instance_url¶
Get the url of the git instance.