MLMatrix (SystemML 0.12.0 API)

java.lang.Object
- org.apache.spark.sql.DataFrame
- - org.apache.sysml.api.MLMatrix

All Implemented Interfaces:

Serializable, org.apache.spark.sql.execution.Queryable
```
public class MLMatrix
extends org.apache.spark.sql.DataFrame
```
Experimental API: Might be discontinued in future release This class serves four purposes: 1. It allows SystemML to fit nicely in MLPipeline by reducing number of reblocks. 2. It allows users to easily read and write matrices without worrying too much about format, metadata and type of underlying RDDs. 3. It provides mechanism to convert to and from MLLib's BlockedMatrix format 4. It provides off-the-shelf library for Distributed Blocked Matrix and reduces learning curve for using SystemML. However, it is important to know that it is easy to abuse this off-the-shelf library and think it as replacement to writing DML, which it is not. It does not provide any optimization between calls. A simple example of the optimization that is conveniently skipped is: (t(m) %*% m)). Also, note that this library is not thread-safe. The operator precedence is not exactly same as DML (as the precedence is enforced by scala compiler), so please use appropriate brackets to enforce precedence. import org.apache.sysml.api.{MLContext, MLMatrix} val ml = new MLContext(sc) val mat1 = ml.read(sqlContext, "V_small.csv", "csv") val mat2 = ml.read(sqlContext, "W_small.mtx", "binary") val result = mat1.transpose() %*% mat2 result.write("Result_small.mtx", "text")

See Also:

Serialized Form

Field Summary

Fields
Modifier and Type Field and Description

protected MatrixCharacteristics mc

protected MLContext ml

Fields
Modifier and Type	Field and Description
`protected MatrixCharacteristics`	`mc`
`protected MLContext`	`ml`

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`MLMatrix(org.apache.spark.sql.DataFrame df, MatrixCharacteristics mc, MLContext ml)`
`protected`	`MLMatrix(org.apache.spark.sql.SQLContext sqlContext, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan, MLContext ml)`
`protected`	`MLMatrix(org.apache.spark.sql.SQLContext sqlContext, org.apache.spark.sql.SQLContext.QueryExecution queryExecution, MLContext ml)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`MLMatrix`	`$bang$eq(Double scalar)`
`MLMatrix`	`$bang$eq(MLMatrix that)`
`MLMatrix`	`$div(Double scalar)`
`MLMatrix`	`$div(MLMatrix that)`
`MLMatrix`	`$eq$eq(Double scalar)`
`MLMatrix`	`$eq$eq(MLMatrix that)`
`MLMatrix`	`$greater(Double scalar)`
`MLMatrix`	`$greater(MLMatrix that)`
`MLMatrix`	`$greater$eq(Double scalar)`
`MLMatrix`	`$greater$eq(MLMatrix that)`
`MLMatrix`	`$less(Double scalar)`
`MLMatrix`	`$less(MLMatrix that)`
`MLMatrix`	`$less$eq(Double scalar)`
`MLMatrix`	`$less$eq(MLMatrix that)`
`MLMatrix`	`$minus(Double scalar)`
`MLMatrix`	`$minus(MLMatrix that)`
`MLMatrix`	`$percent$div$percent(MLMatrix that)`
`MLMatrix`	`$percent$percent(MLMatrix that)`
`MLMatrix`	`$percent$times$percent(MLMatrix that)`
`MLMatrix`	`$plus(Double scalar)`
`MLMatrix`	`$plus(MLMatrix that)`
`MLMatrix`	`$times(Double scalar)`
`MLMatrix`	`$times(MLMatrix that)`
`MLMatrix`	`$up(MLMatrix that)`
`MLMatrix`	`add(Double scalar)`
`MLMatrix`	`add(MLMatrix that)`
`int`	`colsPerBlock()`
`MLMatrix`	`divide(Double scalar)`
`MLMatrix`	`divide(MLMatrix that)`
`MLMatrix`	`elementWiseMultiply(Double scalar)`
`MLMatrix`	`elementWiseMultiply(MLMatrix that)`
`MLMatrix`	`exp(MLMatrix that)`
`MLMatrix`	`integerDivision(MLMatrix that)`
`MLMatrix`	`minus(Double scalar)`
`MLMatrix`	`minus(MLMatrix that)`
`MLMatrix`	`modulus(MLMatrix that)`
`MLMatrix`	`multiply(MLMatrix that)`
`long`	`numCols()` Gets or computes the number of columns.
`long`	`numRows()` Gets or computes the number of rows.
`int`	`rowsPerBlock()`
`MLMatrix`	`transpose()`
`void`	`write(String filePath, String format)` Convenient method to write a MLMatrix.

Methods inherited from class org.apache.spark.sql.DataFrame
agg, agg, agg, agg, agg, alias, alias, apply, as, as, as, cache, coalesce, col, collect, collectAsList, collectToPython, columns, count, createJDBCTable, cube, cube, cube, cube, describe, describe, distinct, drop, drop, dropDuplicates, dropDuplicates, dropDuplicates, dtypes, except, explain, explain, explode, explode, filter, filter, first, flatMap, foreach, foreachPartition, groupBy, groupBy, groupBy, groupBy, head, head, inputFiles, insertInto, insertInto, insertIntoJDBC, intersect, isLocal, javaRDD, javaToPython, join, join, join, join, join, join, limit, logicalPlan, map, mapPartitions, na, numericColumns, orderBy, orderBy, orderBy, orderBy, org$apache$spark$sql$DataFrame$$collect, org$apache$spark$sql$DataFrame$$execute$1, org$apache$spark$sql$DataFrame$$rowFunction$1, org$apache$spark$sql$DataFrame$$withPlan, persist, persist, printSchema, queryExecution, randomSplit, randomSplit, randomSplit, rdd, registerTempTable, repartition, repartition, repartition, repartition, repartition, resolve, rollup, rollup, rollup, rollup, sample, sample, save, save, save, save, save, save, saveAsParquetFile, saveAsTable, saveAsTable, saveAsTable, saveAsTable, saveAsTable, saveAsTable, schema, select, select, select, select, selectExpr, selectExpr, show, show, show, show, showString, showString$default$2, sort, sort, sort, sort, sortWithinPartitions, sortWithinPartitions, sortWithinPartitions, sortWithinPartitions, sqlContext, stat, take, takeAsList, toDF, toDF, toDF, toJavaRDD, toJSON, toSchemaRDD, toString, transform, unionAll, unpersist, unpersist, where, where, withColumn, withColumn, withColumnRenamed, withNewExecutionId, write

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

mc
```
protected MatrixCharacteristics mc
```

ml
```
protected MLContext ml
```

Constructor Detail

MLMatrix

protected MLMatrix(org.apache.spark.sql.SQLContext sqlContext,
                   org.apache.spark.sql.catalyst.plans.logical.LogicalPlan logicalPlan,
                   MLContext ml)

MLMatrix

protected MLMatrix(org.apache.spark.sql.SQLContext sqlContext,
                   org.apache.spark.sql.SQLContext.QueryExecution queryExecution,
                   MLContext ml)

MLMatrix

protected MLMatrix(org.apache.spark.sql.DataFrame df,
                   MatrixCharacteristics mc,
                   MLContext ml)
            throws DMLRuntimeException

Throws:: DMLRuntimeException

Method Detail

write

public void write(String filePath,
                  String format)
           throws IOException,
                  DMLException

Convenient method to write a MLMatrix.

Parameters:: filePath - the file path; format - the format
Throws:: IOException - if IOException occurs; DMLException - if DMLException occurs

numRows
```
public long numRows()
             throws IOException,
                    DMLException
```
Gets or computes the number of rows.

Returns:

the number of rows

Throws:

IOException - if IOException occurs

DMLException - if DMLException occurs

numCols
```
public long numCols()
             throws IOException,
                    DMLException
```
Gets or computes the number of columns.

Returns:

the number of columns

Throws:

IOException - if IOException occurs

DMLException - if DMLException occurs

rowsPerBlock
```
public int rowsPerBlock()
```

colsPerBlock
```
public int colsPerBlock()
```

$greater

public MLMatrix $greater(MLMatrix that)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

$less

public MLMatrix $less(MLMatrix that)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

$greater$eq

public MLMatrix $greater$eq(MLMatrix that)
                     throws IOException,
                            DMLException

Throws:: IOException; DMLException

$less$eq

public MLMatrix $less$eq(MLMatrix that)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

$eq$eq

public MLMatrix $eq$eq(MLMatrix that)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

$bang$eq

public MLMatrix $bang$eq(MLMatrix that)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

$up

public MLMatrix $up(MLMatrix that)
             throws IOException,
                    DMLException

Throws:: IOException; DMLException

exp

public MLMatrix exp(MLMatrix that)
             throws IOException,
                    DMLException

Throws:: IOException; DMLException

$plus

public MLMatrix $plus(MLMatrix that)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

add

public MLMatrix add(MLMatrix that)
             throws IOException,
                    DMLException

Throws:: IOException; DMLException

$minus

public MLMatrix $minus(MLMatrix that)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

minus

public MLMatrix minus(MLMatrix that)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

$times

public MLMatrix $times(MLMatrix that)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

elementWiseMultiply

public MLMatrix elementWiseMultiply(MLMatrix that)
                             throws IOException,
                                    DMLException

Throws:: IOException; DMLException

$div

public MLMatrix $div(MLMatrix that)
              throws IOException,
                     DMLException

Throws:: IOException; DMLException

divide

public MLMatrix divide(MLMatrix that)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

$percent$div$percent

public MLMatrix $percent$div$percent(MLMatrix that)
                              throws IOException,
                                     DMLException

Throws:: IOException; DMLException

integerDivision

public MLMatrix integerDivision(MLMatrix that)
                         throws IOException,
                                DMLException

Throws:: IOException; DMLException

$percent$percent

public MLMatrix $percent$percent(MLMatrix that)
                          throws IOException,
                                 DMLException

Throws:: IOException; DMLException

modulus

public MLMatrix modulus(MLMatrix that)
                 throws IOException,
                        DMLException

Throws:: IOException; DMLException

$percent$times$percent

public MLMatrix $percent$times$percent(MLMatrix that)
                                throws IOException,
                                       DMLException

Throws:: IOException; DMLException

multiply

public MLMatrix multiply(MLMatrix that)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

transpose

public MLMatrix transpose()
                   throws IOException,
                          DMLException

Throws:: IOException; DMLException

$plus

public MLMatrix $plus(Double scalar)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

add

public MLMatrix add(Double scalar)
             throws IOException,
                    DMLException

Throws:: IOException; DMLException

$minus

public MLMatrix $minus(Double scalar)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

minus

public MLMatrix minus(Double scalar)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

$times

public MLMatrix $times(Double scalar)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

elementWiseMultiply

public MLMatrix elementWiseMultiply(Double scalar)
                             throws IOException,
                                    DMLException

Throws:: IOException; DMLException

$div

public MLMatrix $div(Double scalar)
              throws IOException,
                     DMLException

Throws:: IOException; DMLException

divide

public MLMatrix divide(Double scalar)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

$greater

public MLMatrix $greater(Double scalar)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

$less

public MLMatrix $less(Double scalar)
               throws IOException,
                      DMLException

Throws:: IOException; DMLException

$greater$eq

public MLMatrix $greater$eq(Double scalar)
                     throws IOException,
                            DMLException

Throws:: IOException; DMLException

$less$eq

public MLMatrix $less$eq(Double scalar)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

$eq$eq

public MLMatrix $eq$eq(Double scalar)
                throws IOException,
                       DMLException

Throws:: IOException; DMLException

$bang$eq

public MLMatrix $bang$eq(Double scalar)
                  throws IOException,
                         DMLException

Throws:: IOException; DMLException

Class MLMatrix

Field Summary

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.sql.DataFrame

Methods inherited from class java.lang.Object

Field Detail

mc

ml

Constructor Detail

MLMatrix

MLMatrix

MLMatrix

Method Detail

write

numRows

numCols

rowsPerBlock

colsPerBlock

$greater

$less

$greater$eq

$less$eq

$eq$eq

$bang$eq

$up

exp

$plus

add

$minus

minus

$times

elementWiseMultiply

$div

divide

$percent$div$percent

integerDivision

$percent$percent

modulus

$percent$times$percent

multiply

transpose

$plus

add

$minus

minus

$times

elementWiseMultiply

$div

divide

$greater

$less

$greater$eq

$less$eq

$eq$eq

$bang$eq