org.apache.sysml.runtime.instructions.spark

Class MatrixIndexingSPInstruction

java.lang.Object

org.apache.sysml.runtime.instructions.Instruction

org.apache.sysml.runtime.instructions.spark.SPInstruction

org.apache.sysml.runtime.instructions.spark.ComputationSPInstruction

org.apache.sysml.runtime.instructions.spark.UnarySPInstruction

org.apache.sysml.runtime.instructions.spark.IndexingSPInstruction

org.apache.sysml.runtime.instructions.spark.MatrixIndexingSPInstruction

public class MatrixIndexingSPInstruction
extends IndexingSPInstruction

This class implements the matrix indexing functionality inside CP. Example instructions: rangeReIndex:mVar1:Var2:Var3:Var4:Var5:mVar6 input=mVar1, output=mVar6, bounds = (Var2,Var3,Var4,Var5) rowindex_lower: Var2, rowindex_upper: Var3 colindex_lower: Var4, colindex_upper: Var5 leftIndex:mVar1:mVar2:Var3:Var4:Var5:Var6:mVar7 triggered by "mVar1[Var3:Var4, Var5:Var6] = mVar2" the result is stored in mVar7

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysml.runtime.instructions.spark.SPInstruction

SPInstruction.SPINSTRUCTION_TYPE

Nested classes/interfaces inherited from class org.apache.sysml.runtime.instructions.Instruction

Instruction.INSTRUCTION_TYPE

Field Summary

Fields inherited from class org.apache.sysml.runtime.instructions.spark.IndexingSPInstruction

_aggType, colLower, colUpper, rowLower, rowUpper

Fields inherited from class org.apache.sysml.runtime.instructions.spark.ComputationSPInstruction

input1, input2, input3, output

Fields inherited from class org.apache.sysml.runtime.instructions.spark.SPInstruction

_optr, _requiresLabelUpdate, _sptype

Fields inherited from class org.apache.sysml.runtime.instructions.Instruction

beginCol, beginLine, DATATYPE_PREFIX, endCol, endLine, filename, GPU_INST_PREFIX, instOpcode, INSTRUCTION_DELIM, instString, LITERAL_PREFIX, LOG, OPERAND_DELIM, SP_INST_PREFIX, type, VALUETYPE_PREFIX

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MatrixIndexingSPInstruction(org.apache.sysml.runtime.matrix.operators.Operator op, org.apache.sysml.runtime.instructions.cp.CPOperand in, org.apache.sysml.runtime.instructions.cp.CPOperand rl, org.apache.sysml.runtime.instructions.cp.CPOperand ru, org.apache.sysml.runtime.instructions.cp.CPOperand cl, org.apache.sysml.runtime.instructions.cp.CPOperand cu, org.apache.sysml.runtime.instructions.cp.CPOperand out, org.apache.sysml.hops.AggBinaryOp.SparkAggType aggtype, String opcode, String istr)
protected	MatrixIndexingSPInstruction(org.apache.sysml.runtime.matrix.operators.Operator op, org.apache.sysml.runtime.instructions.cp.CPOperand lhsInput, org.apache.sysml.runtime.instructions.cp.CPOperand rhsInput, org.apache.sysml.runtime.instructions.cp.CPOperand rl, org.apache.sysml.runtime.instructions.cp.CPOperand ru, org.apache.sysml.runtime.instructions.cp.CPOperand cl, org.apache.sysml.runtime.instructions.cp.CPOperand cu, org.apache.sysml.runtime.instructions.cp.CPOperand out, org.apache.sysml.lops.LeftIndex.LixCacheType type, String opcode, String istr)

Method Summary

Modifier and Type	Method and Description
static MatrixBlock	inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1, MatrixCharacteristics mcIn, MatrixCharacteristics mcOut, org.apache.sysml.runtime.util.IndexRange ixrange)
static boolean	isMultiBlockLookup(org.apache.spark.api.java.JavaPairRDD<?,?> in, MatrixCharacteristics mcIn, MatrixCharacteristics mcOut, org.apache.sysml.runtime.util.IndexRange ixrange) Indicates if the given index range and input matrix exhibit the following properties: (1) existing hash partitioner, (2) out-of-core input matrix (larger than aggregated memory), (3) aligned indexing range (which does not required aggregation), and (4) the output fits twice in memory (in order to collect the result).
static boolean	isSingleBlockLookup(MatrixCharacteristics mcIn, org.apache.sysml.runtime.util.IndexRange ixrange) Indicates if the given index range only covers a single blocks of the inputs matrix.
void	processInstruction(org.apache.sysml.runtime.controlprogram.context.ExecutionContext ec) This method should be used to execute the instruction.

Methods inherited from class org.apache.sysml.runtime.instructions.spark.IndexingSPInstruction

parseInstruction

Methods inherited from class org.apache.sysml.runtime.instructions.spark.ComputationSPInstruction

getOutputVariableName, updateBinaryOutputMatrixCharacteristics, updateUnaryAggOutputMatrixCharacteristics, updateUnaryOutputMatrixCharacteristics, updateUnaryOutputMatrixCharacteristics

Methods inherited from class org.apache.sysml.runtime.instructions.spark.SPInstruction

getGraphString, getSPInstructionType, postprocessInstruction, preprocessInstruction, requiresLabelUpdate

Methods inherited from class org.apache.sysml.runtime.instructions.Instruction

getBeginColumn, getBeginLine, getEndColumn, getEndLine, getExtendedOpcode, getFilename, getInstID, getLineNum, getOpcode, getType, printMe, setInstID, setLocation, setLocation, setLocation, setLocation, setType, toString, updateInstructionThreadID

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

MatrixIndexingSPInstruction

protected MatrixIndexingSPInstruction(org.apache.sysml.runtime.matrix.operators.Operator op,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand in,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand rl,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand ru,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand cl,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand cu,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand out,
                                      org.apache.sysml.hops.AggBinaryOp.SparkAggType aggtype,
                                      String opcode,
                                      String istr)

MatrixIndexingSPInstruction

protected MatrixIndexingSPInstruction(org.apache.sysml.runtime.matrix.operators.Operator op,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand lhsInput,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand rhsInput,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand rl,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand ru,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand cl,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand cu,
                                      org.apache.sysml.runtime.instructions.cp.CPOperand out,
                                      org.apache.sysml.lops.LeftIndex.LixCacheType type,
                                      String opcode,
                                      String istr)

Method Detail

processInstruction

public void processInstruction(org.apache.sysml.runtime.controlprogram.context.ExecutionContext ec)
                        throws DMLRuntimeException

Description copied from class: Instruction

This method should be used to execute the instruction.

Specified by:

processInstruction in class SPInstruction

Parameters:

ec - execution context

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

inmemoryIndexing

public static MatrixBlock inmemoryIndexing(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> in1,
                                           MatrixCharacteristics mcIn,
                                           MatrixCharacteristics mcOut,
                                           org.apache.sysml.runtime.util.IndexRange ixrange)
                                    throws DMLRuntimeException

Throws:

DMLRuntimeException

isSingleBlockLookup

public static boolean isSingleBlockLookup(MatrixCharacteristics mcIn,
                                          org.apache.sysml.runtime.util.IndexRange ixrange)

Indicates if the given index range only covers a single blocks of the inputs matrix. In this case, we perform a key lookup which is very efficient in case of existing partitioner, especially for out-of-core datasets.

Parameters:

mcIn - matrix characteristics

ixrange - index range

Returns:

true if index range covers a single block of the input matrix

isMultiBlockLookup

public static boolean isMultiBlockLookup(org.apache.spark.api.java.JavaPairRDD<?,?> in,
                                         MatrixCharacteristics mcIn,
                                         MatrixCharacteristics mcOut,
                                         org.apache.sysml.runtime.util.IndexRange ixrange)

Indicates if the given index range and input matrix exhibit the following properties: (1) existing hash partitioner, (2) out-of-core input matrix (larger than aggregated memory), (3) aligned indexing range (which does not required aggregation), and (4) the output fits twice in memory (in order to collect the result).

Parameters:

in - input matrix

mcIn - input matrix characteristics

mcOut - output matrix characteristics

ixrange - index range

Returns:

true if index range requires a multi-block lookup