Cluster module¶
Abstract base class for a compute cluster. A cluster can also be a single machine. This class needs to be implemented and extended for specific cloud providers.
-
class
Cluster.Cluster¶ Bases:
abc.ABCAbstract base class for cloud clusters. It defines a generic interface to implement
-
daskscheduler¶ a reference to the Dask scheduler process started on the cluster
- Type
Popen
-
daskworker¶ a reference to the Dask worker process started on the cluster
- Type
Popen
-
configfile¶ A JSON configuration file containing the required parameters for this class.
- Type
str
-
platform¶ The type of platform or provider
- Type
str
-
abstract
getCoresPN() → int¶ Get the number of cores per node in this cluster. Assumes a heterogenous cluster.
-
abstract
getHosts()¶ Get the list of hosts in this cluster.
-
abstract
getHostsCSV()¶ Get a comma separated list of hosts in this cluster.
-
abstract
getState()¶ Get the cluster state.
-
abstract
parseConfig(cfDict: dict)¶ Parse the cluster configuration. This might contain parameters that are required by specific cloud providers.
- Parameters
cfDict (dict) – Dictionary containing this cluster parameterized settings.
-
abstract
readConfig()¶ Read the cluster configuration.
-
setDaskScheduler(proc: subprocess.Popen)¶ Save the Popen process for dask-scheduler
- Parameters
proc (Popen) – The dask-scheduler process
- Returns
proc – The dask-scheduler process
- Return type
Popen
-
setDaskWorker(proc: subprocess.Popen)¶ Save the Popen process for dask-worker
- Parameters
proc (Popen) – The dask-worker process
- Returns
proc – The dask-worker process
- Return type
Popen
-
abstract
setState()¶ Set the cluster state.
-
abstract
start()¶ Start the cluster.
-
abstract
terminate()¶ Terminate the cluster.
-
terminateDaskScheduler()¶ Kills the dask-scheduler process if it hasn’t terminated yet.
-
terminateDaskWorker()¶ Kills the dask-worker process if it hasn’t terminated yet.
-