Package org.apache.sysds.resource

Class ResourceCompiler

java.lang.Object

org.apache.sysds.resource.ResourceCompiler

public class ResourceCompiler
extends Object

This class does full or partial program recompilation based on given runtime program. It uses the methods provided by hops.recompile.Recompiler). It keeps a state of the current recompilation phase in order to decide when to do full recompilation and when not.

Field Summary

Fields

Modifier and Type	Field	Description
static long	DEFAULT_DRIVER_MEMORY
static int	DEFAULT_DRIVER_THREADS
static long	DEFAULT_EXECUTOR_MEMORY
static int	DEFAULT_EXECUTOR_THREADS
static int	DEFAULT_NUMBER_EXECUTORS

Constructor Summary

Constructors

Constructor	Description
ResourceCompiler()

Method Summary

Modifier and Type	Method	Description
static Program	compile(String filePath, Map<String,String> args)
static Program	compile(String filePath, Map<String,String> args, HashMap<String,String> replaceVars)
static Program	doFullRecompilation(Program program)	Recompiling a given program for resource optimization.
static void	replaceFilename(DMLProgram dmlp, HashMap<String,String> replaceVars)
static void	setSingleNodeResourceConfigs(long nodeMemory, int nodeCores)	Sets resource configurations for executions in single-node mode including the hardware configurations for the node running the CP.
static void	setSparkClusterResourceConfigs(long driverMemory, int driverCores, int numExecutors, long executorMemory, int executorCores)	Sets resource configurations for executions in hybrid mode including the hardware configurations for the node running the CP and the worker nodes running Spark executors

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DRIVER_MEMORY

public static final long DEFAULT_DRIVER_MEMORY

See Also:

Constant Field Values

DEFAULT_DRIVER_THREADS

public static final int DEFAULT_DRIVER_THREADS

See Also:

Constant Field Values

DEFAULT_EXECUTOR_MEMORY

public static final long DEFAULT_EXECUTOR_MEMORY

See Also:

Constant Field Values

DEFAULT_EXECUTOR_THREADS

public static final int DEFAULT_EXECUTOR_THREADS

See Also:

Constant Field Values

DEFAULT_NUMBER_EXECUTORS

public static final int DEFAULT_NUMBER_EXECUTORS

See Also:

Constant Field Values

Constructor Detail

ResourceCompiler

public ResourceCompiler()

Method Detail

compile

public static Program compile(String filePath,
                              Map<String,String> args)
                       throws IOException

Throws:

IOException

compile

public static Program compile(String filePath,
                              Map<String,String> args,
                              HashMap<String,String> replaceVars)
                       throws IOException

Throws:

IOException

replaceFilename

public static void replaceFilename(DMLProgram dmlp,
                                   HashMap<String,String> replaceVars)

doFullRecompilation

public static Program doFullRecompilation(Program program)

Recompiling a given program for resource optimization. This method should always be called after setting the target resources to InfrastructureAnalyzer and SparkExecutionContext

Parameters:

program - program to be recompiled

Returns:

the recompiled program as a new Program instance

setSingleNodeResourceConfigs

public static void setSingleNodeResourceConfigs(long nodeMemory,
                                                int nodeCores)

Sets resource configurations for executions in single-node mode including the hardware configurations for the node running the CP.

Parameters:

nodeMemory - memory budget for the node running CP

nodeCores - number of CPU cores for the node running CP

setSparkClusterResourceConfigs

public static void setSparkClusterResourceConfigs(long driverMemory,
                                                  int driverCores,
                                                  int numExecutors,
                                                  long executorMemory,
                                                  int executorCores)

Sets resource configurations for executions in hybrid mode including the hardware configurations for the node running the CP and the worker nodes running Spark executors

Parameters:

driverMemory - memory budget for the node running CP

driverCores - number of CPU cores for the node running CP

numExecutors - number of nodes in cluster

executorMemory - memory budget for the nodes running executors

executorCores - number of CPU cores for the nodes running executors