| Modifier and Type | Method and Description |
|---|---|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
MLContextConversionUtil.dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
Convert a
DataFrame to a
JavaPairRDD<MatrixIndexes, MatrixBlock> binary-block matrix. |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
MLContextConversionUtil.dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame,
MatrixMetadata matrixMetadata)
Convert a
DataFrame to a
JavaPairRDD<MatrixIndexes, MatrixBlock> binary-block matrix. |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
MLContextConversionUtil.matrixObjectToBinaryBlocks(org.apache.sysml.runtime.controlprogram.caching.MatrixObject matrixObject,
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sparkExecutionContext)
Convert a
MatrixObject to a
JavaPairRDD<MatrixIndexes, MatrixBlock>. |
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
Matrix.toBinaryBlocks()
Obtain the matrix as a
JavaPairRDD<MatrixIndexes, MatrixBlock> |
| Modifier and Type | Method and Description |
|---|---|
static MatrixBlock |
MLContextConversionUtil.binaryBlocksToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks,
MatrixMetadata matrixMetadata)
Convert a
JavaPairRDD<MatrixIndexes, MatrixBlock> to a
MatrixBlock |
static org.apache.sysml.runtime.controlprogram.caching.MatrixObject |
MLContextConversionUtil.binaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks)
Convert a
JavaPairRDD<MatrixIndexes, MatrixBlock> to a
MatrixObject. |
static org.apache.sysml.runtime.controlprogram.caching.MatrixObject |
MLContextConversionUtil.binaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks,
MatrixMetadata matrixMetadata)
Convert a
JavaPairRDD<MatrixIndexes, MatrixBlock> to a
MatrixObject. |
| Constructor and Description |
|---|
Matrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks,
MatrixMetadata matrixMetadata)
Create a Matrix, specifying the SystemML binary-block matrix and its
metadata.
|
| Modifier and Type | Method and Description |
|---|---|
MatrixBlock |
CompressedMatrixBlock.aggregateBinaryOperations(MatrixIndexes m1Index,
MatrixBlock m1Value,
MatrixIndexes m2Index,
MatrixBlock m2Value,
MatrixBlock result,
org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op) |
MatrixValue |
CompressedMatrixBlock.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn) |
MatrixValue |
CompressedMatrixBlock.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn,
boolean inCP) |
void |
CompressedMatrixBlock.ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
MatrixIndexes ix1,
double scalar,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock) |
| Modifier and Type | Method and Description |
|---|---|
scala.Tuple2<MatrixIndexes,MatrixBlock> |
BuiltinNarySPInstruction.PadBlocksFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
AppendGSPInstruction.ShiftMatrix.call(scala.Tuple2<MatrixIndexes,MatrixBlock> kv) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
AppendGAlignedSPInstruction.ShiftColumnIndex.call(scala.Tuple2<MatrixIndexes,MatrixBlock> kv) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
SpoofSPInstruction.ReplicateRightFactorFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunctionInMem.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRExpandFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDMapGroupedAggFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformDecodeExpandFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> in) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
AppendGSPInstruction.MergeWithShiftedBlocks.call(scala.Tuple2<MatrixIndexes,scala.Tuple2<Iterable<MatrixBlock>,Iterable<MatrixBlock>>> kv) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunction.call(scala.Tuple2<MatrixIndexes,scala.Tuple2<MatrixBlock,MatrixBlock>> arg0) |
protected org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
CSVReblockSPInstruction.processMatrixCSVReblockInstruction(org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
MatrixCharacteristics mcOut) |
| Modifier and Type | Method and Description |
|---|---|
scala.Tuple2<MatrixIndexes,MatrixBlock> |
BuiltinNarySPInstruction.PadBlocksFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
MatrixBlock |
AggregateUnarySPInstruction.RDDUAggFunction2.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
AppendGSPInstruction.ShiftMatrix.call(scala.Tuple2<MatrixIndexes,MatrixBlock> kv) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
AppendGAlignedSPInstruction.ShiftColumnIndex.call(scala.Tuple2<MatrixIndexes,MatrixBlock> kv) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
SpoofSPInstruction.ReplicateRightFactorFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunctionInMem.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRExpandFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDMapGroupedAggFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
MatrixBlock |
ParameterizedBuiltinSPInstruction.RDDMapGroupedAggFunction2.call(scala.Tuple2<MatrixIndexes,MatrixBlock> arg0) |
scala.Tuple2<Long,FrameBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformDecodeFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> in) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformDecodeExpandFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> in) |
scala.Tuple2<MatrixIndexes,MatrixBlock> |
AppendGSPInstruction.MergeWithShiftedBlocks.call(scala.Tuple2<MatrixIndexes,scala.Tuple2<Iterable<MatrixBlock>,Iterable<MatrixBlock>>> kv) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
ParameterizedBuiltinSPInstruction.RDDRemoveEmptyFunction.call(scala.Tuple2<MatrixIndexes,scala.Tuple2<MatrixBlock,MatrixBlock>> arg0) |
static MatrixBlock |
MatrixIndexingSPInstruction.inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1,
MatrixCharacteristics mcIn,
MatrixCharacteristics mcOut,
org.apache.sysml.runtime.util.IndexRange ixrange) |
void |
ParameterizedBuiltinSPInstruction.setOutputCharacteristicsForGroupedAgg(MatrixCharacteristics mc1,
MatrixCharacteristics mcOut,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> out) |
| Modifier and Type | Method and Description |
|---|---|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop,
int numPartitions,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
FrameRDDConverterUtils.binaryBlockToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> input,
MatrixCharacteristics mcIn,
MatrixCharacteristics mcOut) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.binaryCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input,
MatrixCharacteristics mcOut,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> |
SparkUtils.cacheBinaryCellRDD(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
RDDConverterUtils.BinaryCellToBinaryBlockFunction.call(Iterator<scala.Tuple2<MatrixIndexes,MatrixCell>> arg0) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtilsExt.coordinateMatrixToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
MatrixCharacteristics mcIn,
boolean outputEmptyBlocks)
Example usage:
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtilsExt.coordinateMatrixToBinaryBlock(org.apache.spark.SparkContext sc,
org.apache.spark.mllib.linalg.distributed.CoordinateMatrix input,
MatrixCharacteristics mcIn,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.copyBinaryBlockMatrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Creates a partitioning-preserving deep copy of the input matrix RDD, where
the indexes and values are copied.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.copyBinaryBlockMatrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deep)
Creates a partitioning-preserving copy of the input matrix RDD.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mc,
boolean hasHeader,
String delim,
boolean fill,
double fillValue) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaRDD<String> input,
MatrixCharacteristics mcOut,
boolean hasHeader,
String delim,
boolean fill,
double fillValue)
Example usage:
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
MatrixCharacteristics mc,
boolean containsID,
boolean isVector) |
static scala.Tuple2<MatrixIndexes,MatrixBlock> |
SparkUtils.fromIndexedMatrixBlock(org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue in) |
static List<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
SparkUtils.fromIndexedMatrixBlock(List<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> in) |
static Pair<MatrixIndexes,MatrixBlock> |
SparkUtils.fromIndexedMatrixBlockToPair(org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue in) |
static List<Pair<MatrixIndexes,MatrixBlock>> |
SparkUtils.fromIndexedMatrixBlockToPair(List<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.getEmptyBlockRDD(org.apache.spark.api.java.JavaSparkContext sc,
MatrixCharacteristics mc)
Creates an RDD of empty blocks according to the given matrix characteristics.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deepCopyCombiner)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
int numPartitions,
boolean deepCopyCombiner)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeRowsByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,org.apache.sysml.runtime.instructions.spark.data.RowMatrixBlock> in)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in2,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
long rlen,
long clen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByIx(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> ixmap,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByValMemSort(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen,
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
org.apache.sysml.runtime.matrix.operators.ReorgOperator r_op)
This function collects and sorts value column in memory and then broadcasts it.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
long clen2,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortIndexesByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
boolean asc,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortIndexesByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean asc,
long rlen,
long clen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
int numPartitions,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> |
RDDAggregateUtils.sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> |
RDDAggregateUtils.sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in,
int numParts) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.textCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
MatrixCharacteristics mcOut,
boolean outputEmptyBlocks,
org.apache.sysml.runtime.io.FileFormatPropertiesMM mmProps) |
| Modifier and Type | Method and Description |
|---|---|
static org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue |
SparkUtils.toIndexedMatrixBlock(MatrixIndexes ix,
MatrixBlock mb) |
| Modifier and Type | Method and Description |
|---|---|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.aggByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop,
int numPartitions,
boolean deepCopyCombiner) |
static MatrixBlock |
RDDAggregateUtils.aggStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.sysml.runtime.matrix.operators.AggregateOperator aop)
Single block aggregation over pair rdds with corrections for numerical stability.
|
static org.apache.spark.api.java.JavaRDD<String> |
RDDConverterUtils.binaryBlockToCsv(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mcIn,
org.apache.sysml.runtime.io.FileFormatPropertiesCSV props,
boolean strict) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
RDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SparkSession sparkSession,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc,
boolean toVector) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
RDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SQLContext sqlContext,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc,
boolean toVector)
Deprecated.
|
static org.apache.spark.api.java.JavaRDD<org.apache.spark.ml.feature.LabeledPoint> |
RDDConverterUtils.binaryBlockToLabeledPoints(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Converter from binary block rdd to rdd of labeled points.
|
static org.apache.spark.api.java.JavaRDD<String> |
RDDConverterUtils.binaryBlockToTextCell(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
MatrixCharacteristics mc) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDConverterUtils.binaryCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input,
MatrixCharacteristics mcOut,
boolean outputEmptyBlocks) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> |
SparkUtils.cacheBinaryCellRDD(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input) |
Iterator<scala.Tuple2<MatrixIndexes,MatrixBlock>> |
RDDConverterUtils.BinaryCellToBinaryBlockFunction.call(Iterator<scala.Tuple2<MatrixIndexes,MatrixCell>> arg0) |
static MatrixCharacteristics |
SparkUtils.computeMatrixCharacteristics(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixCell> input)
Utility to compute dimensions and non-zeros in a given RDD of binary cells.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.copyBinaryBlockMatrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Creates a partitioning-preserving deep copy of the input matrix RDD, where
the indexes and values are copied.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
SparkUtils.copyBinaryBlockMatrix(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deep)
Creates a partitioning-preserving copy of the input matrix RDD.
|
static long |
SparkUtils.getNonZeros(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input) |
static org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
MatrixCharacteristics mcIn) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlockLongIndex(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
MatrixCharacteristics mcIn) |
static double |
RDDAggregateUtils.max(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deepCopyCombiner)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
int numPartitions,
boolean deepCopyCombiner)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.mergeRowsByKey(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,org.apache.sysml.runtime.instructions.spark.data.RowMatrixBlock> in)
Merges disjoint data of all blocks per key.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in2,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in2,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
long rlen,
long clen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByIx(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> ixmap,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByIx(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> ixmap,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByValMemSort(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen,
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
org.apache.sysml.runtime.matrix.operators.ReorgOperator r_op)
This function collects and sorts value column in memory and then broadcasts it.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByValMemSort(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
int brlen,
int bclen,
org.apache.sysml.runtime.controlprogram.context.SparkExecutionContext sec,
org.apache.sysml.runtime.matrix.operators.ReorgOperator r_op)
This function collects and sorts value column in memory and then broadcasts it.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
long clen2,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortDataByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> data,
boolean asc,
long rlen,
long clen,
long clen2,
int brlen,
int bclen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortIndexesByVal(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> val,
boolean asc,
long rlen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDSortUtils.sortIndexesByVals(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean asc,
long rlen,
long clen,
int brlen) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
RDDAggregateUtils.sumByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in,
int numPartitions,
boolean deepCopyCombiner) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> |
RDDAggregateUtils.sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in) |
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> |
RDDAggregateUtils.sumCellsByKeyStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,Double> in,
int numParts) |
static MatrixBlock |
RDDAggregateUtils.sumStable(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in) |
static org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue |
SparkUtils.toIndexedMatrixBlock(scala.Tuple2<MatrixIndexes,MatrixBlock> in) |
| Modifier and Type | Method and Description |
|---|---|
void |
CombineMR.InnerReducer.reduce(MatrixIndexes indexes,
Iterator<TaggedMatrixValue> values,
org.apache.hadoop.mapred.OutputCollector<MatrixIndexes,WeightedPair> out,
org.apache.hadoop.mapred.Reporter reporter) |
| Modifier and Type | Method and Description |
|---|---|
void |
CombineMR.InnerReducer.reduce(MatrixIndexes indexes,
Iterator<TaggedMatrixValue> values,
org.apache.hadoop.mapred.OutputCollector<MatrixIndexes,WeightedPair> out,
org.apache.hadoop.mapred.Reporter reporter) |
| Modifier and Type | Method and Description |
|---|---|
MatrixIndexes |
MatrixIndexes.setIndexes(long r,
long c) |
MatrixIndexes |
MatrixIndexes.setIndexes(MatrixIndexes that) |
| Modifier and Type | Method and Description |
|---|---|
Pair<MatrixIndexes,MatrixBlock> |
TextCellToRowBlockConverter.next() |
Pair<MatrixIndexes,MatrixBlock> |
BinaryCellToRowBlockConverter.next() |
Pair<MatrixIndexes,WeightedPair> |
AddDummyWeightConverter.next() |
Pair<MatrixIndexes,MatrixBlock> |
BinaryBlockToRowBlockConverter.next() |
Pair<MatrixIndexes,MatrixCell> |
TextToBinaryCellConverter.next() |
Pair<MatrixIndexes,MatrixCell> |
BinaryBlockToBinaryCellConverter.next() |
| Modifier and Type | Method and Description |
|---|---|
MatrixBlock |
MatrixBlock.aggregateBinaryOperations(MatrixIndexes m1Index,
MatrixBlock m1,
MatrixIndexes m2Index,
MatrixBlock m2,
MatrixBlock ret,
org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op) |
MatrixValue |
MatrixBlock.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn) |
MatrixValue |
WeightedCell.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int brlen,
int bclen,
MatrixIndexes indexesIn) |
abstract MatrixValue |
MatrixValue.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int brlen,
int bclen,
MatrixIndexes indexesIn) |
MatrixValue |
MatrixCell.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int brlen,
int bclen,
MatrixIndexes indexesIn) |
MatrixValue |
CM_N_COVCell.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int brlen,
int bclen,
MatrixIndexes indexesIn) |
MatrixValue |
MatrixBlock.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn,
boolean inCP) |
abstract MatrixValue |
MatrixValue.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn,
boolean inCP) |
MatrixValue |
MatrixCell.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn,
boolean inCP) |
MatrixValue |
CM_N_COVCell.aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
MatrixValue result,
int blockingFactorRow,
int blockingFactorCol,
MatrixIndexes indexesIn,
boolean inCP) |
int |
MatrixIndexes.compare(MatrixIndexes m1,
MatrixIndexes m2) |
int |
MatrixIndexes.compareTo(MatrixIndexes other) |
void |
BinaryBlockToRowBlockConverter.convert(MatrixIndexes k1,
MatrixBlock v1) |
void |
BinaryBlockToTextCellConverter.convert(MatrixIndexes k1,
MatrixBlock v1)
Before calling convert, please make sure to setBlockSize(brlen, bclen);
|
void |
BinaryBlockToBinaryCellConverter.convert(MatrixIndexes k1,
MatrixBlock v1) |
void |
BinaryCellToRowBlockConverter.convert(MatrixIndexes k1,
MatrixCell v1) |
void |
BinaryCellToTextConverter.convert(MatrixIndexes k1,
MatrixCell v1) |
void |
WeightedCellToSortInputConverter.convert(MatrixIndexes k1,
WeightedCell v1) |
void |
MatrixBlock.ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
MatrixIndexes ix1,
double scalarThat,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock)
Specific ctable case of ctable(seq(...),X), where X is the only
matrix input.
|
abstract void |
MatrixValue.ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
MatrixIndexes ix1,
double scalar_that,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock) |
void |
MatrixCell.ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
MatrixIndexes ix1,
double scalarThat,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock) |
void |
CM_N_COVCell.ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
MatrixIndexes ix1,
double scalarThat,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock) |
static MatrixBlock |
OperationsOnMatrixValues.matMult(MatrixIndexes indexes1,
MatrixBlock value1,
MatrixIndexes indexes2,
MatrixBlock value2,
MatrixIndexes indexesOut,
MatrixBlock valueOut,
org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op) |
static void |
OperationsOnMatrixValues.performAggregateUnary(MatrixIndexes indexesIn,
MatrixValue valueIn,
MatrixIndexes indexesOut,
MatrixValue valueOut,
org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
int brlen,
int bclen) |
static void |
OperationsOnMatrixValues.performCtable(MatrixIndexes indexesIn1,
MatrixValue valueIn1,
double scalarIn2,
boolean left,
int brlen,
CTableMap resultMap,
MatrixBlock resultBlock,
org.apache.sysml.runtime.matrix.operators.Operator op) |
static void |
OperationsOnMatrixValues.performCtable(MatrixIndexes indexesIn1,
MatrixValue valueIn1,
double scalarIn2,
double scalarIn3,
CTableMap resultMap,
MatrixBlock resultBlock,
org.apache.sysml.runtime.matrix.operators.Operator op) |
static void |
OperationsOnMatrixValues.performCtable(MatrixIndexes indexesIn1,
MatrixValue valueIn1,
double scalarIn2,
MatrixIndexes indexesIn3,
MatrixValue valueIn3,
CTableMap resultMap,
MatrixBlock resultBlock,
org.apache.sysml.runtime.matrix.operators.Operator op) |
static void |
OperationsOnMatrixValues.performCtable(MatrixIndexes indexesIn1,
MatrixValue valueIn1,
MatrixIndexes indexesIn2,
MatrixValue valueIn2,
double scalarIn3,
CTableMap resultMap,
MatrixBlock resultBlock,
org.apache.sysml.runtime.matrix.operators.Operator op) |
static void |
OperationsOnMatrixValues.performCtable(MatrixIndexes indexesIn1,
MatrixValue valueIn1,
MatrixIndexes indexesIn2,
MatrixValue valueIn2,
MatrixIndexes indexesIn3,
MatrixValue valueIn3,
CTableMap resultMap,
MatrixBlock resultBlock,
org.apache.sysml.runtime.matrix.operators.Operator op) |
static void |
OperationsOnMatrixValues.performReorg(MatrixIndexes indexesIn,
MatrixValue valueIn,
MatrixIndexes indexesOut,
MatrixValue valueOut,
org.apache.sysml.runtime.matrix.operators.ReorgOperator op,
int startRow,
int startColumn,
int length) |
static void |
OperationsOnMatrixValues.performZeroOut(MatrixIndexes indexesIn,
MatrixValue valueIn,
MatrixIndexes indexesOut,
MatrixValue valueOut,
org.apache.sysml.runtime.util.IndexRange range,
boolean complementary) |
static void |
LibMatrixAgg.recomputeIndexes(MatrixBlock out,
org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
int brlen,
int bclen,
MatrixIndexes ix)
Recompute outputs (e.g., maxindex or minindex) according to block indexes from MR.
|
static void |
LibMatrixOuterAgg.resetOutputMatrix(MatrixIndexes in1Ix,
MatrixBlock in1Val,
MatrixIndexes outIx,
MatrixBlock outVal,
org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)
ReSet output matrix
|
MatrixIndexes |
MatrixIndexes.setIndexes(MatrixIndexes that) |
| Constructor and Description |
|---|
MatrixIndexes(MatrixIndexes indexes) |
TaggedMatrixIndexes(MatrixIndexes ix,
byte t) |
Copyright © 2018 The Apache Software Foundation. All rights reserved.