org.apache.sysml.api.mlcontext

Class MLContextUtil

java.lang.Object

org.apache.sysml.api.mlcontext.MLContextUtil

public final class MLContextUtil
extends Object

Utility class containing methods for working with the MLContext API.

Field Summary

Fields

Modifier and Type	Field and Description
static Class[]	ALL_SUPPORTED_DATA_TYPES All data types supported by the MLContext API
static Class[]	BASIC_DATA_TYPES Basic data types supported by the MLContext API
static Class[]	COMPLEX_DATA_TYPES Complex data types supported by the MLContext API

Constructor Summary

Constructors

Constructor and Description
MLContextUtil()

Method Summary

Modifier and Type	Method and Description
static void	checkInputParameterType(String parameterName, Object parameterValue) Verify that the type of input parameter value is supported.
static void	checkInputValueType(String name, Object value) Verify that the type of input value is supported.
static void	checkInputValueTypes(Map<String,Object> inputs) Verify that the types of input values are supported.
static Map<String,String>	convertInputParametersForParser(Map<String,Object> basicInputParameterMap, ScriptType scriptType) Converts non-string basic input parameter values to strings to pass to the parser.
static org.apache.sysml.runtime.instructions.cp.Data	convertInputType(String parameterName, Object parameterValue) Convert input types to internal SystemML representations
static org.apache.sysml.runtime.instructions.cp.Data	convertInputType(String parameterName, Object parameterValue, Metadata metadata) Convert input types to internal SystemML representations
static String	createHistoryForScript(Script script, long when) Generate a String history entry for a script.
static void	deleteRemoveVariableInstructions(org.apache.sysml.runtime.controlprogram.Program progam) Delete the 'remove variable' instructions from a runtime program.
static String	determineOutputTypeAsString(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable, String outputName) Obtain a symbol table output type as a String
static String	displayInputs(String name, Map<String,Object> map, org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable) Obtain a display of script inputs.
static String	displayMap(String mapName, Map<String,Object> map) Display the keys and values in a Map
static String	displayOutputs(Set<String> outputNames, org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable) Obtain a display of the script outputs.
static String	displayOutputs(String name, Set<String> outputNames, org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable) Obtain a display of the script outputs.
static String	displayScriptHistory(List<String> scriptHistory) Generate a String listing of the script execution history.
static String	displaySet(String setName, Set<String> set) Display the values in a Set
static String	displaySymbolTable(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable) Display the keys and values in the symbol table
static String	displaySymbolTable(String name, org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable) Display the keys and values in the symbol table
static boolean	doesDataFrameLookLikeMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df) Examine the DataFrame schema to determine whether the data appears to be a matrix.
static boolean	doesSymbolTableContainFrameObject(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable, String variableName) Determine if the symbol table contains a FrameObject with the given variable name.
static boolean	doesSymbolTableContainMatrixObject(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable, String variableName) Determine if the symbol table contains a MatrixObject with the given variable name.
static String	getBasicTypeString(Object object) Obtain the SystemML scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)
static org.apache.spark.api.java.JavaSparkContext	getJavaSparkContext(MLContext mlContext) Obtain the Java Spark Context
static org.apache.spark.SparkContext	getSparkContext(MLContext mlContext) Obtain the Spark Context
static boolean	isBasicType(Object object) Is the object one of the supported basic data types? (Integer, Boolean, Double, String)
static boolean	isComplexType(Object object) Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, BinaryBlockMatrix, Matrix, double[][], MatrixBlock, URL)
static boolean	isCSVLineAllNumbers(String line) If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)
static boolean	isSparkVersionSupported(String sparkVersion) Determine whether the Spark version is supported.
static String	quotedString(String str) Return a double-quoted string with inner single and double quotes escaped.
static void	setCompilerConfig() Set SystemML compiler configuration properties for MLContext
static void	setConfig(String configFilePath) Set SystemML configuration properties based on a configuration file.
static void	setDefaultConfig() Set default SystemML configuration properties.
static void	verifySparkVersionSupported(org.apache.spark.SparkContext sc) Check that the Spark version is supported.
static String	welcomeMessage() The SystemML welcome message

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BASIC_DATA_TYPES

public static final Class[] BASIC_DATA_TYPES

Basic data types supported by the MLContext API

COMPLEX_DATA_TYPES

public static final Class[] COMPLEX_DATA_TYPES

Complex data types supported by the MLContext API

ALL_SUPPORTED_DATA_TYPES

public static final Class[] ALL_SUPPORTED_DATA_TYPES

All data types supported by the MLContext API

Constructor Detail

MLContextUtil

public MLContextUtil()

Method Detail

isSparkVersionSupported

public static boolean isSparkVersionSupported(String sparkVersion)

Determine whether the Spark version is supported.

Parameters:

sparkVersion - Spark version string (ie, "1.5.0").

Returns:

true if Spark version supported; otherwise false.

verifySparkVersionSupported

public static void verifySparkVersionSupported(org.apache.spark.SparkContext sc)

Check that the Spark version is supported. If it isn't supported, throw an MLContextException.

Parameters:

sc - SparkContext

Throws:

MLContextException - thrown if Spark version isn't supported

setDefaultConfig

public static void setDefaultConfig()

Set default SystemML configuration properties.

setConfig

public static void setConfig(String configFilePath)

Set SystemML configuration properties based on a configuration file.

Parameters:

configFilePath - Path to configuration file.

Throws:

MLContextException - if configuration file was not found or a parse exception occurred

setCompilerConfig

public static void setCompilerConfig()

Set SystemML compiler configuration properties for MLContext

checkInputValueTypes

public static void checkInputValueTypes(Map<String,Object> inputs)

Verify that the types of input values are supported.

Parameters:

inputs - Map of String/Object pairs

Throws:

MLContextException - if an input value type is not supported

checkInputValueType

public static void checkInputValueType(String name,
                                       Object value)

Verify that the type of input value is supported.

Parameters:

name - The name of the input

value - The value of the input

Throws:

MLContextException - if the input value type is not supported

checkInputParameterType

public static void checkInputParameterType(String parameterName,
                                           Object parameterValue)

Verify that the type of input parameter value is supported.

Parameters:

parameterName - The name of the input parameter

parameterValue - The value of the input parameter

Throws:

MLContextException - if the input parameter value type is not supported

isBasicType

public static boolean isBasicType(Object object)

Is the object one of the supported basic data types? (Integer, Boolean, Double, String)

Parameters:

object - the object type to be examined

Returns:

true if type is a basic data type; otherwise false.

getBasicTypeString

public static String getBasicTypeString(Object object)

Obtain the SystemML scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)

Parameters:

object - the object type to be examined

Returns:

a String representing the type as a SystemML scalar value type

isComplexType

public static boolean isComplexType(Object object)

Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, BinaryBlockMatrix, Matrix, double[][], MatrixBlock, URL)

Parameters:

object - the object type to be examined

Returns:

true if type is a complex data type; otherwise false.

convertInputParametersForParser

public static Map<String,String> convertInputParametersForParser(Map<String,Object> basicInputParameterMap,
                                                                 ScriptType scriptType)

Converts non-string basic input parameter values to strings to pass to the parser.

Parameters:

basicInputParameterMap - map of input parameters

scriptType - ScriptType.DML or ScriptType.PYDML

Returns:

map of String/String name/value pairs

convertInputType

public static org.apache.sysml.runtime.instructions.cp.Data convertInputType(String parameterName,
                                                                             Object parameterValue)

Convert input types to internal SystemML representations

Parameters:

parameterName - The name of the input parameter

parameterValue - The value of the input parameter

Returns:

input in SystemML data representation

convertInputType

public static org.apache.sysml.runtime.instructions.cp.Data convertInputType(String parameterName,
                                                                             Object parameterValue,
                                                                             Metadata metadata)

Convert input types to internal SystemML representations

Parameters:

parameterName - The name of the input parameter

parameterValue - The value of the input parameter

metadata - matrix/frame metadata

Returns:

input in SystemML data representation

isCSVLineAllNumbers

public static boolean isCSVLineAllNumbers(String line)

If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)

Parameters:

line - a line of the RDD

Returns:

true if all the csv-separated values are numbers, false otherwise

doesDataFrameLookLikeMatrix

public static boolean doesDataFrameLookLikeMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)

Examine the DataFrame schema to determine whether the data appears to be a matrix.

Parameters:

df - the DataFrame

Returns:

true if the DataFrame appears to be a matrix, false otherwise

quotedString

public static String quotedString(String str)

Return a double-quoted string with inner single and double quotes escaped.

Parameters:

str - the original string

Returns:

double-quoted string with inner single and double quotes escaped

displayMap

public static String displayMap(String mapName,
                                Map<String,Object> map)

Display the keys and values in a Map

Parameters:

mapName - the name of the map

map - Map of String keys and Object values

Returns:

the keys and values in the Map as a String

displaySet

public static String displaySet(String setName,
                                Set<String> set)

Display the values in a Set

Parameters:

setName - the name of the Set

set - Set of String values

Returns:

the values in the Set as a String

displaySymbolTable

public static String displaySymbolTable(String name,
                                        org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable)

Display the keys and values in the symbol table

Parameters:

name - the name of the symbol table

symbolTable - the LocalVariableMap

Returns:

the keys and values in the symbol table as a String

displaySymbolTable

public static String displaySymbolTable(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable)

Display the keys and values in the symbol table

Parameters:

symbolTable - the LocalVariableMap

Returns:

the keys and values in the symbol table as a String

determineOutputTypeAsString

public static String determineOutputTypeAsString(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable,
                                                 String outputName)

Obtain a symbol table output type as a String

Parameters:

symbolTable - the symbol table

outputName - the name of the output variable

Returns:

the symbol table output type for a variable as a String

displayInputs

public static String displayInputs(String name,
                                   Map<String,Object> map,
                                   org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable)

Obtain a display of script inputs.

Parameters:

name - the title to display for the inputs

map - the map of inputs

symbolTable - the symbol table

Returns:

the script inputs represented as a String

displayOutputs

public static String displayOutputs(String name,
                                    Set<String> outputNames,
                                    org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable)

Obtain a display of the script outputs.

Parameters:

name - the title to display for the outputs

outputNames - the names of the output variables

symbolTable - the symbol table

Returns:

the script outputs represented as a String

displayOutputs

public static String displayOutputs(Set<String> outputNames,
                                    org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable)

Obtain a display of the script outputs.

Parameters:

outputNames - the names of the output variables

symbolTable - the symbol table

Returns:

the script outputs represented as a String

welcomeMessage

public static String welcomeMessage()

The SystemML welcome message

Returns:

the SystemML welcome message

createHistoryForScript

public static String createHistoryForScript(Script script,
                                            long when)

Generate a String history entry for a script.

Parameters:

script - the script

when - when the script was executed

Returns:

a script history entry as a String

displayScriptHistory

public static String displayScriptHistory(List<String> scriptHistory)

Generate a String listing of the script execution history.

Parameters:

scriptHistory - the list of script history entries

Returns:

the listing of the script execution history as a String

getSparkContext

public static org.apache.spark.SparkContext getSparkContext(MLContext mlContext)

Obtain the Spark Context

Parameters:

mlContext - the SystemML MLContext

Returns:

the Spark Context

getJavaSparkContext

public static org.apache.spark.api.java.JavaSparkContext getJavaSparkContext(MLContext mlContext)

Obtain the Java Spark Context

Parameters:

mlContext - the SystemML MLContext

Returns:

the Java Spark Context

doesSymbolTableContainFrameObject

public static boolean doesSymbolTableContainFrameObject(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable,
                                                        String variableName)

Determine if the symbol table contains a FrameObject with the given variable name.

Parameters:

symbolTable - the LocalVariableMap

variableName - the variable name

Returns:

true if the variable in the symbol table is a FrameObject, false otherwise.

doesSymbolTableContainMatrixObject

public static boolean doesSymbolTableContainMatrixObject(org.apache.sysml.runtime.controlprogram.LocalVariableMap symbolTable,
                                                         String variableName)

Determine if the symbol table contains a MatrixObject with the given variable name.

Parameters:

symbolTable - the LocalVariableMap

variableName - the variable name

Returns:

true if the variable in the symbol table is a MatrixObject, false otherwise.

deleteRemoveVariableInstructions

public static void deleteRemoveVariableInstructions(org.apache.sysml.runtime.controlprogram.Program progam)

Delete the 'remove variable' instructions from a runtime program.

Parameters:

progam - runtime program