org.apache.sysds.runtime.controlprogram.paramserv.dp

Class DataPartitionerSparkMapper

java.lang.Object

org.apache.sysds.runtime.controlprogram.paramserv.dp.DataPartitionerSparkMapper

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>

public class DataPartitionerSparkMapper
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>, Serializable

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DataPartitionerSparkMapper(Statement.PSScheme scheme, int workersNum, SparkExecutionContext sec, int numEntries)

Method Summary

Modifier and Type	Method and Description
Iterator<scala.Tuple2<Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>>	call(scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>> input) Do data partitioning

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataPartitionerSparkMapper

public DataPartitionerSparkMapper(Statement.PSScheme scheme,
                                  int workersNum,
                                  SparkExecutionContext sec,
                                  int numEntries)

Method Detail

call

public Iterator<scala.Tuple2<Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>> call(scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>> input)
                                                                                              throws Exception

Do data partitioning

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>,Integer,scala.Tuple2<Long,scala.Tuple2<MatrixBlock,MatrixBlock>>>

Parameters:

input - RowBlockID => (features, labels)

Returns:

WorkerID => (rowBlockID, (single row features, single row labels))

Throws:

Exception - Some exception