org.apache.sysds.api.mlcontext

Class ScriptExecutor

java.lang.Object

org.apache.sysds.api.mlcontext.ScriptExecutor

public class ScriptExecutor
extends Object

ScriptExecutor executes a DML or PYDML Script object using SystemDS. This is accomplished by calling the execute(org.apache.sysds.api.mlcontext.Script) method.

Script execution via the MLContext API typically consists of the following steps:

1. Language Steps
   1. Parse script into program
   2. Live variable analysis
   3. Validate program
2. HOP (High-Level Operator) Steps
   1. Construct HOP DAGs
   2. Static rewrites
   3. Intra-/Inter-procedural analysis
   4. Dynamic rewrites
   5. Compute memory estimates
   6. Rewrite persistent reads and writes (MLContext-specific)
3. LOP (Low-Level Operator) Steps
   1. Contruct LOP DAGs
   2. Generate runtime program
   3. Execute runtime program
   4. Dynamic recompilation

Modifications to these steps can be accomplished by subclassing ScriptExecutor. For more information, please see the execute(org.apache.sysds.api.mlcontext.Script) method.

Constructor Summary

Constructors

Constructor and Description
ScriptExecutor() ScriptExecutor constructor.
ScriptExecutor(DMLConfig config) ScriptExecutor constructor, where the configuration properties are passed in.

Method Summary

Modifier and Type	Method and Description
void	compile(Script script)
void	compile(Script script, boolean performHOPRewrites) Compile a DML or PYDML script.
MLResults	execute(Script script) Execute a DML or PYDML script.
DMLConfig	getConfig() Obtain the SystemDS configuration properties.
DMLProgram	getDmlProgram() Obtain the program
DMLTranslator	getDmlTranslator() Obtain the translator
ExecutionContext	getExecutionContext() Obtain the execution context
MLContext.ExecutionType	getExecutionType() Obtain the current execution environment.
Program	getRuntimeProgram() Obtain the runtime program
Script	getScript() Obtain the Script object associated with this ScriptExecutor
boolean	isMaintainSymbolTable() Obtain whether or not all values should be maintained in the symbol table after execution.
void	setConfig(DMLConfig config) Set the SystemDS configuration properties.
void	setExecutionType(MLContext.ExecutionType executionType) Set the execution environment.
void	setExplain(boolean explain) Whether or not an explanation of the DML/PYDML program should be output to standard output.
void	setExplainLevel(MLContext.ExplainLevel explainLevel) Set the level of program explanation that should be displayed if explain is set to true.
void	setForceGPU(boolean enabled) Whether or not to force GPU usage.
void	setGPU(boolean enabled) Whether or not to enable GPU usage.
void	setInit(boolean init) Whether or not to initialize the scratch_space, bufferpool, etc.
void	setMaintainSymbolTable(boolean maintainSymbolTable) Set whether or not all values should be maintained in the symbol table after execution.
void	setStatistics(boolean statistics) Whether or not statistics about the DML/PYDML program should be output to standard output.
void	setStatisticsMaxHeavyHitters(int maxHeavyHitters) Set the maximum number of heavy hitters to display with statistics.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ScriptExecutor

public ScriptExecutor()

ScriptExecutor constructor.

ScriptExecutor

public ScriptExecutor(DMLConfig config)

ScriptExecutor constructor, where the configuration properties are passed in.

Parameters:

config - the configuration properties to use by the ScriptExecutor

Method Detail

compile

public void compile(Script script)

compile

public void compile(Script script,
                    boolean performHOPRewrites)

Compile a DML or PYDML script. This will help analysis of DML programs that have dynamic recompilation flag set to false without actually executing it. This is broken down into the following primary methods:

1. setup(Script)
2. parseScript()
3. liveVariableAnalysis()
4. validateScript()
5. constructHops()
6. rewriteHops()
7. rewritePersistentReadsAndWrites()
8. constructLops()
9. generateRuntimeProgram()
10. showExplanation()
11. countCompiledMRJobsAndSparkInstructions()
12. initializeCachingAndScratchSpace()
13. cleanupRuntimeProgram()

Parameters:

script - the DML or PYDML script to compile

performHOPRewrites - should perform static rewrites, perform intra-/inter-procedural analysis to propagate size information into functions and apply dynamic rewrites

execute

public MLResults execute(Script script)

Execute a DML or PYDML script. This is broken down into the following primary methods:

1. compile(Script)
2. createAndInitializeExecutionContext()
3. executeRuntimeProgram()
4. cleanupAfterExecution()

Parameters:

script - the DML or PYDML script to execute

Returns:

the results as a MLResults object

setConfig

public void setConfig(DMLConfig config)

Set the SystemDS configuration properties.

Parameters:

config - The configuration properties

getDmlProgram

public DMLProgram getDmlProgram()

Obtain the program

Returns:

the program

getDmlTranslator

public DMLTranslator getDmlTranslator()

Obtain the translator

Returns:

the translator

getRuntimeProgram

public Program getRuntimeProgram()

Obtain the runtime program

Returns:

the runtime program

getExecutionContext

public ExecutionContext getExecutionContext()

Obtain the execution context

Returns:

the execution context

getScript

public Script getScript()

Obtain the Script object associated with this ScriptExecutor

Returns:

the Script object associated with this ScriptExecutor

setExplain

public void setExplain(boolean explain)

Whether or not an explanation of the DML/PYDML program should be output to standard output.

Parameters:

explain - true if explanation should be output, false otherwise

setStatistics

public void setStatistics(boolean statistics)

Whether or not statistics about the DML/PYDML program should be output to standard output.

Parameters:

statistics - true if statistics should be output, false otherwise

setStatisticsMaxHeavyHitters

public void setStatisticsMaxHeavyHitters(int maxHeavyHitters)

Set the maximum number of heavy hitters to display with statistics.

Parameters:

maxHeavyHitters - the maximum number of heavy hitters

isMaintainSymbolTable

public boolean isMaintainSymbolTable()

Obtain whether or not all values should be maintained in the symbol table after execution.

Returns:

true if all values should be maintained in the symbol table, false otherwise

setMaintainSymbolTable

public void setMaintainSymbolTable(boolean maintainSymbolTable)

Set whether or not all values should be maintained in the symbol table after execution.

Parameters:

maintainSymbolTable - true if all values should be maintained in the symbol table, false otherwise

setInit

public void setInit(boolean init)

Whether or not to initialize the scratch_space, bufferpool, etc. Note that any redundant initialize (e.g., multiple scripts from one MLContext) clears existing files from the scratch space and buffer pool.

Parameters:

init - true if should initialize, false otherwise

setExplainLevel

public void setExplainLevel(MLContext.ExplainLevel explainLevel)

Set the level of program explanation that should be displayed if explain is set to true.

Parameters:

explainLevel - the level of program explanation

setGPU

public void setGPU(boolean enabled)

Whether or not to enable GPU usage.

Parameters:

enabled - true if enabled, false otherwise

setForceGPU

public void setForceGPU(boolean enabled)

Whether or not to force GPU usage.

Parameters:

enabled - true if enabled, false otherwise

getConfig

public DMLConfig getConfig()

Obtain the SystemDS configuration properties.

Returns:

the configuration properties

getExecutionType

public MLContext.ExecutionType getExecutionType()

Obtain the current execution environment.

Returns:

the execution environment

setExecutionType

public void setExecutionType(MLContext.ExecutionType executionType)

Set the execution environment.

Parameters:

executionType - the execution environment