Package org.apache.sysds.api.jmlc

Class Connection

java.lang.Object

org.apache.sysds.api.jmlc.Connection

All Implemented Interfaces:

Closeable, AutoCloseable

public class Connection
extends Object
implements Closeable

Interaction with SystemDS using the JMLC (Java Machine Learning Connector) API is initiated with a Connection object. The JMLC API is designed after JDBC. A DML script is precompiled by calling the prepareScript(String, String[], String[]) method or the prepareScript(String, Map, String[], String[]) method on the Connection object, which returns a PreparedScript object. Note that this is similar to calling a prepareStatement method on a JDBC Connection object.

Following this, input variable data is passed to the script by calling the setFrame, setMatrix, and setScalar methods of the PreparedScript object. The script is executed via PreparedScript's executeScript method, which returns a ResultVariables object, which is similar to a JDBC ResultSet. Data can be read from a ResultVariables object by calling its getFrame and getMatrix methods.

Constructor Summary

Constructors

Constructor	Description
Connection()	Connection constructor, the starting point for any other JMLC API calls.
Connection(CompilerConfig.ConfigType... cconfigs)	Connection constructor, the starting point for any other JMLC API calls.
Connection(DMLConfig dmlconfig)	Connection constructor, the starting point for any other JMLC API calls.
Connection(DMLConfig dmlconfig, CompilerConfig.ConfigType... cconfigs)	Connection constructor, the starting point for any other JMLC API calls.

Method Summary

Modifier and Type	Method	Description
void	close()	Close connection to SystemDS, which clears the thread-local DML and compiler configurations.
double[][]	convertToDoubleMatrix(InputStream input, int rows, int cols)	Converts an input stream of a string matrix in textcell format into a dense double array.
double[][]	convertToDoubleMatrix(InputStream input, int rows, int cols, String format)	Converts an input stream of a string matrix in csv or textcell format into a dense double array.
double[][]	convertToDoubleMatrix(String input, int rows, int cols)	Converts an input string representation of a matrix in textcell format into a dense double array.
double[][]	convertToDoubleMatrix(String input, String meta)	Converts an input string representation of a matrix in csv or textcell format into a dense double array.
FrameBlock	convertToFrame(InputStream input, int rows, int cols)	Converts an input stream of a string frame in textcell format into a frame block.
FrameBlock	convertToFrame(InputStream input, int rows, int cols, String format)	Converts an input stream of a frame in csv or textcell format into a frame block.
FrameBlock	convertToFrame(InputStream input, String meta)	Converts an input stream of a string frame in csv or textcell format into a frame block.
FrameBlock	convertToFrame(String input, int rows, int cols)	Converts an input string representation of a frame in textcell format into a frame block.
FrameBlock	convertToFrame(String input, String meta)	Converts an input string representation of a frame in csv or textcell format into a frame block.
MatrixBlock	convertToMatrix(InputStream input, int rows, int cols)	Converts an input stream of a string matrix in text format into a matrix block.
MatrixBlock	convertToMatrix(InputStream input, int rows, int cols, String format)	Converts an input stream of a string matrix in csv or text format into a matrix block.
MatrixBlock	convertToMatrix(InputStream input, String meta)	Converts an input stream of a string matrix in csv or textcell format into a matrix block.
MatrixBlock	convertToMatrix(String input, int rows, int cols)	Converts an input string representation of a matrix in textcell format into a matrix block.
MatrixBlock	convertToMatrix(String input, String meta)	Converts an input string representation of a matrix in csv or textcell format into a matrix block.
String[][]	convertToStringFrame(InputStream input, int rows, int cols)	Converts an input stream of a string frame in textcell format into a dense string array.
String[][]	convertToStringFrame(InputStream input, int rows, int cols, String format)	Converts an input stream of a string frame in csv or text format into a dense string array.
String[][]	convertToStringFrame(String input, int rows, int cols)	Converts an input stream of a string frame in textcell format into a dense string array.
String[][]	convertToStringFrame(String input, String meta)	Converts an input string representation of a frame in csv or textcell format into a dense string array.
void	gatherMemStats(boolean stats)	Sets a boolean flag indicating if memory profiling statistics should be gathered.
PreparedScript	prepareScript(String script, String[] inputs, String[] outputs)	Prepares (precompiles) a script and registers input and output variables.
PreparedScript	prepareScript(String script, Map<String,String> args, String[] inputs, String[] outputs)	Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
PreparedScript	prepareScript(String script, Map<String,String> nsscripts, Map<String,String> args, String[] inputs, String[] outputs)	Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
double[][]	readDoubleMatrix(String fname)	Reads an input matrix in arbitrary format from HDFS into a dense double array.
double[][]	readDoubleMatrix(String fname, Types.FileFormat fmt, long rows, long cols, int blen, long nnz)	Reads an input matrix in arbitrary format from HDFS into a dense double array.
String	readScript(String fname)	Read a DML or PyDML file as a string.
String[][]	readStringFrame(String fname)	Reads an input frame in arbitrary format from HDFS into a dense string array.
String[][]	readStringFrame(String fname, Types.FileFormat fmt, long rows, long cols)	Reads an input frame in arbitrary format from HDFS into a dense string array.
FrameBlock	readTransformMetaDataFromFile(String metapath)	Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.
FrameBlock	readTransformMetaDataFromFile(String spec, String metapath)	Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.
FrameBlock	readTransformMetaDataFromFile(String spec, String metapath, String colDelim)	Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.
FrameBlock	readTransformMetaDataFromPath(String metapath)	Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.
FrameBlock	readTransformMetaDataFromPath(String spec, String metapath)	Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.
FrameBlock	readTransformMetaDataFromPath(String spec, String metapath, String colDelim)	Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.
void	setConfigTypes(boolean activate, CompilerConfig.ConfigType... cconfigs)	Sets compiler configs.
void	setLineage(boolean lt)	Sets a boolean flag indicating if lineage trace should be captured
void	setStatistics(boolean stats)	Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Connection

public Connection()

Connection constructor, the starting point for any other JMLC API calls.

Connection

public Connection(CompilerConfig.ConfigType... cconfigs)

Connection constructor, the starting point for any other JMLC API calls. This variant allows to enable a set of boolean compiler configurations.

Parameters:

cconfigs - one or many boolean compiler configurations to enable.

Connection

public Connection(DMLConfig dmlconfig,
                  CompilerConfig.ConfigType... cconfigs)

Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration and enable a set of boolean compiler configurations.

Parameters:

dmlconfig - a dml configuration.

cconfigs - one or many boolean compiler configurations to enable.

Connection

public Connection(DMLConfig dmlconfig)

Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration.

Parameters:

dmlconfig - a dml configuration.

Method Detail

setConfigTypes

public void setConfigTypes(boolean activate,
                           CompilerConfig.ConfigType... cconfigs)

Sets compiler configs.

Parameters:

activate - activate or disable

cconfigs - the configs to set

setStatistics

public void setStatistics(boolean stats)

Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"

Parameters:

stats - boolean value with true indicating statistics should be gathered

setLineage

public void setLineage(boolean lt)

Sets a boolean flag indicating if lineage trace should be captured

Parameters:

lt - boolean value with true indicating lineage should be captured

gatherMemStats

public void gatherMemStats(boolean stats)

Sets a boolean flag indicating if memory profiling statistics should be gathered. The option is false by default.

Parameters:

stats - boolean value with true indicating memory statistics should be gathered

prepareScript

public PreparedScript prepareScript(String script,
                                    String[] inputs,
                                    String[] outputs)

Prepares (precompiles) a script and registers input and output variables.

Parameters:

script - string representing the DML or PyDML script

inputs - string array of input variables to register

outputs - string array of output variables to register

Returns:

PreparedScript object representing the precompiled script

prepareScript

public PreparedScript prepareScript(String script,
                                    Map<String,String> args,
                                    String[] inputs,
                                    String[] outputs)

Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.

Parameters:

script - string representing the DML or PyDML script

args - map of input parameters ($) and their values

inputs - string array of input variables to register

outputs - string array of output variables to register

Returns:

PreparedScript object representing the precompiled script

prepareScript

public PreparedScript prepareScript(String script,
                                    Map<String,String> nsscripts,
                                    Map<String,String> args,
                                    String[] inputs,
                                    String[] outputs)

Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.

Parameters:

script - string representing of the DML or PyDML script

nsscripts - map (name, script) of the DML or PyDML namespace scripts

args - map of input parameters ($) and their values

inputs - string array of input variables to register

outputs - string array of output variables to register

Returns:

PreparedScript object representing the precompiled script

close

public void close()

Close connection to SystemDS, which clears the thread-local DML and compiler configurations.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

readScript

public String readScript(String fname)
                  throws IOException

Read a DML or PyDML file as a string.

Parameters:

fname - the filename of the script

Returns:

string content of the script file

Throws:

IOException - if IOException occurs

readDoubleMatrix

public double[][] readDoubleMatrix(String fname)
                            throws IOException

Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.

Parameters:

fname - the filename of the input matrix

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

readDoubleMatrix

public double[][] readDoubleMatrix(String fname,
                                   Types.FileFormat fmt,
                                   long rows,
                                   long cols,
                                   int blen,
                                   long nnz)
                            throws IOException

Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.

Parameters:

fname - the filename of the input matrix

fmt - file format

rows - number of rows in the matrix

cols - number of columns in the matrix

blen - block length

nnz - number of non-zero values, -1 indicates unknown

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

convertToDoubleMatrix

public double[][] convertToDoubleMatrix(String input,
                                        String meta)
                                 throws IOException

Converts an input string representation of a matrix in csv or textcell format into a dense double array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - string matrix in csv or textcell format

meta - string representing SystemDS matrix metadata in JSON format

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

convertToDoubleMatrix

public double[][] convertToDoubleMatrix(String input,
                                        int rows,
                                        int cols)
                                 throws IOException

Converts an input string representation of a matrix in textcell format into a dense double array.

Parameters:

input - string matrix in textcell format

rows - number of rows in the matrix

cols - number of columns in the matrix

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

convertToDoubleMatrix

public double[][] convertToDoubleMatrix(InputStream input,
                                        int rows,
                                        int cols)
                                 throws IOException

Converts an input stream of a string matrix in textcell format into a dense double array.

Parameters:

input - InputStream to a string matrix in textcell format

rows - number of rows in the matrix

cols - number of columns in the matrix

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

convertToDoubleMatrix

public double[][] convertToDoubleMatrix(InputStream input,
                                        int rows,
                                        int cols,
                                        String format)
                                 throws IOException

Converts an input stream of a string matrix in csv or textcell format into a dense double array.

Parameters:

input - InputStream to a string matrix in csv or textcell format

rows - number of rows in the matrix

cols - number of columns in the matrix

format - input format of the given stream

Returns:

matrix as a two-dimensional double array

Throws:

IOException - if IOException occurs

convertToMatrix

public MatrixBlock convertToMatrix(String input,
                                   String meta)
                            throws IOException

Converts an input string representation of a matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - string matrix in csv or textcell format

meta - string representing SystemDS matrix metadata in JSON format

Returns:

matrix as a matrix block

Throws:

IOException - if IOException occurs

convertToMatrix

public MatrixBlock convertToMatrix(InputStream input,
                                   String meta)
                            throws IOException

Converts an input stream of a string matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - InputStream to a string matrix in csv or textcell format

meta - string representing SystemDS matrix metadata in JSON format

Returns:

matrix as a matrix block

Throws:

IOException - if IOException occurs

convertToMatrix

public MatrixBlock convertToMatrix(String input,
                                   int rows,
                                   int cols)
                            throws IOException

Converts an input string representation of a matrix in textcell format into a matrix block.

Parameters:

input - string matrix in textcell format

rows - number of rows in the matrix

cols - number of columns in the matrix

Returns:

matrix as a matrix block

Throws:

IOException - if IOException occurs

convertToMatrix

public MatrixBlock convertToMatrix(InputStream input,
                                   int rows,
                                   int cols)
                            throws IOException

Converts an input stream of a string matrix in text format into a matrix block.

Parameters:

input - InputStream to a string matrix in text format

rows - number of rows in the matrix

cols - number of columns in the matrix

Returns:

matrix as a matrix block

Throws:

IOException - if IOException occurs

convertToMatrix

public MatrixBlock convertToMatrix(InputStream input,
                                   int rows,
                                   int cols,
                                   String format)
                            throws IOException

Converts an input stream of a string matrix in csv or text format into a matrix block.

Parameters:

input - InputStream to a string matrix in csv or text format

rows - number of rows in the matrix

cols - number of columns in the matrix

format - input format of the given stream

Returns:

matrix as a matrix block

Throws:

IOException - if IOException occurs

readStringFrame

public String[][] readStringFrame(String fname)
                           throws IOException

Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.

Parameters:

fname - the filename of the input frame

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

readStringFrame

public String[][] readStringFrame(String fname,
                                  Types.FileFormat fmt,
                                  long rows,
                                  long cols)
                           throws IOException

Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.

Parameters:

fname - the filename of the input frame

fmt - file format type

rows - number of rows in the frame

cols - number of columns in the frame

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

convertToStringFrame

public String[][] convertToStringFrame(String input,
                                       String meta)
                                throws IOException

Converts an input string representation of a frame in csv or textcell format into a dense string array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - string frame in csv or textcell format

meta - string representing SystemDS frame metadata in JSON format

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

convertToStringFrame

public String[][] convertToStringFrame(String input,
                                       int rows,
                                       int cols)
                                throws IOException

Converts an input stream of a string frame in textcell format into a dense string array.

Parameters:

input - string frame in textcell format

rows - number of rows in the frame

cols - number of columns in the frame

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

convertToStringFrame

public String[][] convertToStringFrame(InputStream input,
                                       int rows,
                                       int cols)
                                throws IOException

Converts an input stream of a string frame in textcell format into a dense string array.

Parameters:

input - InputStream to a string frame in text format

rows - number of rows in the frame

cols - number of columns in the frame

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

convertToStringFrame

public String[][] convertToStringFrame(InputStream input,
                                       int rows,
                                       int cols,
                                       String format)
                                throws IOException

Converts an input stream of a string frame in csv or text format into a dense string array.

Parameters:

input - InputStream to a string frame in csv or text format

rows - number of rows in the frame

cols - number of columns in the frame

format - input format of the given stream

Returns:

frame as a two-dimensional string array

Throws:

IOException - if IOException occurs

convertToFrame

public FrameBlock convertToFrame(String input,
                                 String meta)
                          throws IOException

Converts an input string representation of a frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - string frame in csv or textcell format

meta - string representing SystemDS frame metadata in JSON format

Returns:

frame as a frame block

Throws:

IOException - if IOException occurs

convertToFrame

public FrameBlock convertToFrame(InputStream input,
                                 String meta)
                          throws IOException

Converts an input stream of a string frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.

Parameters:

input - InputStream to a string frame in csv or textcell format

meta - string representing SystemDS frame metadata in JSON format

Returns:

frame as a frame block

Throws:

IOException - if IOException occurs

convertToFrame

public FrameBlock convertToFrame(String input,
                                 int rows,
                                 int cols)
                          throws IOException

Converts an input string representation of a frame in textcell format into a frame block.

Parameters:

input - string frame in textcell format

rows - number of rows in the frame

cols - number of columns in the frame

Returns:

frame as a frame block

Throws:

IOException - if IOException occurs

convertToFrame

public FrameBlock convertToFrame(InputStream input,
                                 int rows,
                                 int cols)
                          throws IOException

Converts an input stream of a string frame in textcell format into a frame block.

Parameters:

input - InputStream to a string frame in textcell format

rows - number of rows in the frame

cols - number of columns in the frame

Returns:

frame as a frame block

Throws:

IOException - if IOException occurs

convertToFrame

public FrameBlock convertToFrame(InputStream input,
                                 int rows,
                                 int cols,
                                 String format)
                          throws IOException

Converts an input stream of a frame in csv or textcell format into a frame block.

Parameters:

input - InputStream to a string frame in csv or textcell format

rows - number of rows in the frame

cols - number of columns in the frame

format - input format of the given stream

Returns:

frame as a frame block

Throws:

IOException - if IOException occurs

readTransformMetaDataFromFile

public FrameBlock readTransformMetaDataFromFile(String metapath)
                                         throws IOException

Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.

Parameters:

metapath - hdfs file path to meta data directory

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs

readTransformMetaDataFromFile

public FrameBlock readTransformMetaDataFromFile(String spec,
                                                String metapath)
                                         throws IOException

Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.

Parameters:

spec - transform specification as json string

metapath - hdfs file path to meta data directory

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs

readTransformMetaDataFromFile

public FrameBlock readTransformMetaDataFromFile(String spec,
                                                String metapath,
                                                String colDelim)
                                         throws IOException

Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.

Parameters:

spec - transform specification as json string

metapath - hdfs file path to meta data directory

colDelim - separator for processing column names in the meta data file 'column.names'

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs

readTransformMetaDataFromPath

public FrameBlock readTransformMetaDataFromPath(String metapath)
                                         throws IOException

Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.

Parameters:

metapath - resource path to meta data directory

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs

readTransformMetaDataFromPath

public FrameBlock readTransformMetaDataFromPath(String spec,
                                                String metapath)
                                         throws IOException

Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.

Parameters:

spec - transform specification as json string

metapath - resource path to meta data directory

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs

readTransformMetaDataFromPath

public FrameBlock readTransformMetaDataFromPath(String spec,
                                                String metapath,
                                                String colDelim)
                                         throws IOException

Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.

Parameters:

spec - transform specification as json string

metapath - resource path to meta data directory

colDelim - separator for processing column names in the meta data file 'column.names'

Returns:

FrameBlock object representing transform metadata

Throws:

IOException - if IOException occurs