cluster_tasks module

Various cluster related functions wrapped as Prefect Tasks

cluster_tasks.cluster_init(config) → cloudflow.cluster.Cluster.Cluster

Create a new Cluster object using the ClusterFactory

Parameters

config (string) –

Returns

newcluster – Object returned will be a sub-class of Cluster

Return type

Cluster

cluster_tasks.cluster_start(cluster)

Start the cluster

Parameters

cluster (Cluster) –

cluster_tasks.cluster_terminate(cluster)

Terminate the cluster

Parameters

cluster (Cluster) –

cluster_tasks.dask_client_close(daskclient: distributed.client.Client)

Close the Dask Client

Parameters

daskclient (Dask distributed.Client) –

cluster_tasks.push_pyEnv(cluster)

Push any local python packages to the newly started cluster and pip3 install them. This is needed when local changes to the plotting routines exist. The cluster may not have the updated version.

Parameters

cluster (Cluster) –

Notes

Use python3 setup.py sdist to create the distributable source package to upload.

cluster_tasks.start_dask(cluster) → distributed.client.Client

Create a Dask cluster on the remote or local host

Parameters

cluster (Cluster) –

Returns

daskclient – The connected Dask Client to use for submitting jobs

Return type

Dask distributed.Client