Package org.apache.sysds.resource.enumeration

Class PruneBasedEnumerator

java.lang.Object

org.apache.sysds.resource.enumeration.Enumerator

org.apache.sysds.resource.enumeration.PruneBasedEnumerator

public class PruneBasedEnumerator
extends Enumerator

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.resource.enumeration.Enumerator

Enumerator.Builder, Enumerator.EnumerationStrategy, Enumerator.OptimizationStrategy

Field Summary

Fields inherited from class org.apache.sysds.resource.enumeration.Enumerator

COST_DELTA_FRACTION, DEFAULT_MAX_EXECUTORS, DEFAULT_MIN_EXECUTORS

Constructor Summary

Constructors

Constructor	Description
PruneBasedEnumerator(Enumerator.Builder builder)

Method Summary

Modifier and Type	Method	Description
static long	combineHash(long executorMemory, int cores)	Ensures unique mapping for a combination of node memory and number of executor cores due to the discrete nature of the node memory given in bytes.
ArrayList<Integer>	estimateRangeExecutors(int driverCores, long executorMemory, int executorCores)	Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations
boolean	evaluateSingleNodeExecution(long driverMemory, int cores)
static boolean	hasSparkInstructions(Program program)	Checks for Spark instruction in the given runtime program.
static boolean	isInvalidConfiguration(double[] estimates)
void	preprocessing()	Called once to enumerate the search space for VM instances for driver or executor nodes.
void	processing()	Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space.

Methods inherited from class org.apache.sysds.resource.enumeration.Enumerator

getCostsWeightFactor, getDriverSpace, getEnumStrategy, getExecutorSpace, getInstances, getMaxPrice, getMaxTime, getOptimalSolution, getOptStrategy, postprocessing, setCostsWeightFactor, setCpuQuota, setDriverSpace, setExecutorSpace, setMinPrice, setMinTime, updateOptimalSolution

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PruneBasedEnumerator

public PruneBasedEnumerator(Enumerator.Builder builder)

Method Detail

preprocessing

public void preprocessing()

Description copied from class: Enumerator

Called once to enumerate the search space for VM instances for driver or executor nodes. These instances are being represented as

Specified by:

preprocessing in class Enumerator

processing

public void processing()

Description copied from class: Enumerator

Called once after preprocessing to fill the pool with optimal solutions by parsing the enumerated search space. Within its execution the number of potential executor nodes is being estimated (enumerated) dynamically for each parsed executor instance.

Overrides:

processing in class Enumerator

evaluateSingleNodeExecution

public boolean evaluateSingleNodeExecution(long driverMemory,
                                           int cores)

Specified by:

evaluateSingleNodeExecution in class Enumerator

estimateRangeExecutors

public ArrayList<Integer> estimateRangeExecutors(int driverCores,
                                                 long executorMemory,
                                                 int executorCores)

Description copied from class: Enumerator

Estimates the minimum and maximum number of executors based on given VM instance characteristics, the enumeration strategy and the user-defined configurations

Specified by:

estimateRangeExecutors in class Enumerator

Parameters:

driverCores - CPU cores for the currently evaluated driver node

executorMemory - memory of currently evaluated executor node

executorCores - CPU cores of currently evaluated executor node

Returns:

- [min, max]

combineHash

public static long combineHash(long executorMemory,
                               int cores)

Ensures unique mapping for a combination of node memory and number of executor cores due to the discrete nature of the node memory given in bytes. The smallest margin for cloud instances would be around 500MB ~ 500*1024^2 bytes, what is by far larger than the maximum number of executors core physically possible.

Parameters:

executorMemory - node memory in bytes

cores - number virtual cores (physical threads) for the node

Returns:

hash value

isInvalidConfiguration

public static boolean isInvalidConfiguration(double[] estimates)

hasSparkInstructions

public static boolean hasSparkInstructions(Program program)

Checks for Spark instruction in the given runtime program. It excludes from the check instructions for reblock operations and for caching since these are not always remove from the runtime program even their outputs are never used and ignored at execution.

Parameters:

program - runtime program

Returns:

boolean to mark if the execution would execute any Spark operation