org.apache.sysml.runtime.compress

Class CompressedMatrixBlock

java.lang.Object

org.apache.sysml.runtime.matrix.data.MatrixValue

org.apache.sysml.runtime.matrix.data.MatrixBlock

org.apache.sysml.runtime.compress.CompressedMatrixBlock

All Implemented Interfaces:

Externalizable, Serializable, Comparable, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, org.apache.sysml.runtime.controlprogram.caching.CacheBlock

public class CompressedMatrixBlock
extends MatrixBlock
implements Externalizable

Experimental version of MatrixBlock that allows a compressed internal representation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompressedMatrixBlock.CompressionStatistics

Nested classes/interfaces inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

MatrixBlock.BlockType, MatrixBlock.SparsityEstimate

Nested classes/interfaces inherited from class org.apache.sysml.runtime.matrix.data.MatrixValue

MatrixValue.CellIndex

Field Summary

Fields

Modifier and Type	Field and Description
protected ArrayList<ColGroup>	_colGroups
protected CompressedMatrixBlock.CompressionStatistics	_stats
static boolean	ALLOW_DDC_ENCODING
static boolean	INVESTIGATE_ESTIMATES
static boolean	MATERIALIZE_ZEROS
static long	MIN_PAR_AGG_THRESHOLD
static boolean	TRANSPOSE_INPUT

Fields inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

clen, DEFAULT_INPLACE_SPARSEBLOCK, DEFAULT_SPARSEBLOCK, denseBlock, diag, estimatedNNzsPerRow, HEADER_SIZE, MAX_SHALLOW_SERIALIZE_OVERHEAD, nonZeros, numGroups, rlen, sparse, sparseBlock, SPARSITY_TURN_POINT, ULTRA_SPARSITY_TURN_POINT

Constructor Summary

Constructors

Constructor and Description
CompressedMatrixBlock()
CompressedMatrixBlock(int rl, int cl, boolean sparse) Main constructor for building a block from scratch.
CompressedMatrixBlock(MatrixBlock mb) "Copy" constructor to populate this compressed block with the uncompressed contents of a conventional block.

Method Summary

Modifier and Type	Method and Description
MatrixValue	aggregateBinaryOperations(MatrixIndexes m1Index, MatrixValue m1Value, MatrixIndexes m2Index, MatrixValue m2Value, MatrixValue result, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixValue	aggregateBinaryOperations(MatrixValue mv1, MatrixValue mv2, MatrixValue result, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixBlock	aggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op, boolean inCP)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn, boolean inCP)
MatrixBlock	appendOperations(MatrixBlock that, MatrixBlock ret)
MatrixBlock	appendOperations(MatrixBlock that, MatrixBlock ret, boolean cbind)
void	appendOperations(MatrixValue v2, ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, int blockRowFactor, int blockColFactor, boolean cbind, boolean m2IsLast, int nextNCol)
MatrixValue	binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue, MatrixValue result)
void	binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype, int k)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op, MatrixBlock weights)
void	compress() Compress the contents of this matrix block.
void	compress(int k) Compress block.
int[]	countNonZerosPerRow(int rl, int ru)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that, MatrixBlock weights)
MatrixBlock	decompress() Decompress block.
MatrixBlock	decompress(int k) Decompress block.
long	estimateCompressedSizeInMemory() Obtain an upper bound on the memory used to store the compressed block.
ArrayList<ColGroup>	getColGroups() Obtain the column groups.
CompressedMatrixBlock.CompressionStatistics	getCompressionStatistics()
Iterator<double[]>	getDenseRowIterator(int rl, int ru)
long	getExactSizeOnDisk() NOTE: The used estimates must be kept consistent with the respective write functions.
Iterator<IJV>	getIterator(int rl, int ru, boolean inclZeros)
Iterator<IJV>	getIterator(int rl, int ru, int cgl, int cgu, boolean inclZeros)
int	getNumColGroups()
Iterator<SparseRow>	getSparseRowIterator(int rl, int ru)
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op) Invocation from CP instructions.
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op, int k)
boolean	hasUncompressedColGroup()
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue newWithCorrection)
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection)
boolean	isCompressed() Obtain whether this block is in compressed form or not.
boolean	isEmptyBlock(boolean safe)
boolean	isSingleUncompressedGroup()
MatrixBlock	leftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
MatrixBlock	leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar, int rl, int cl, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update) Explicitly allow left indexing for scalars.
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val)
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)
MatrixValue	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixValue um, MatrixValue vm, MatrixValue wm, MatrixValue out)
MatrixValue	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixValue um, MatrixValue vm, MatrixValue wm, MatrixValue out, int k)
double	quickGetValue(int r, int c)
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, LongStream nnzInBlock, org.apache.commons.math3.random.Well1024a bigrand, long bSeed) Function to generate a matrix of random numbers.
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, LongStream nnzInBlock, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k) Function to generate a matrix of random numbers.
void	readExternal(ObjectInput is) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.
void	readFields(DataInput in)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, MatrixBlock select)
MatrixValue	reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)
MatrixValue	replaceOperations(MatrixValue result, double pattern, double replacement)
MatrixBlock	rexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
MatrixValue	scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator sop, MatrixValue result)
MatrixBlock	seqOperationsInPlace(double from, double to, double incr)
void	sliceOperations(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, org.apache.sysml.runtime.util.IndexRange range, int rowCut, int colCut, int normalBlockRowFactor, int normalBlockColFactor, int boundaryRlen, int boundaryClen)
MatrixBlock	sliceOperations(int rl, int ru, int cl, int cu, org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret) Method to perform rangeReIndex operation for a given lower and upper bounds in row and column dimensions.
MatrixValue	sortOperations(MatrixValue weights, MatrixValue result)
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalar, double scalar2, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalar, MatrixValue that, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixIndexes ix1, double scalar, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock) Specific ctable case of ctable(seq(...),X), where X is the only matrix input.
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, double scalar, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices.
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, double scalar, MatrixBlock resultBlock) D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap) D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)
void	ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap, MatrixBlock resultBlock)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype, int k)
MatrixBlock	uaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp, org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)
MatrixValue	unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op, MatrixValue result)
void	unaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.UnaryOperator op)
void	write(DataOutput out)
void	writeExternal(ObjectOutput os) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.
MatrixValue	zeroOutOperations(MatrixValue result, org.apache.sysml.runtime.util.IndexRange range, boolean complementary)

Methods inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

allocateAndResetSparseRowsBlock, allocateDenseBlock, allocateDenseBlock, allocateDenseBlockUnsafe, allocateDenseOrSparseBlock, allocateSparseRowsBlock, allocateSparseRowsBlock, appendRow, appendRow, appendToSparse, appendToSparse, appendValue, checkNonZeros, checkSparseRows, cleanupBlock, compactEmptyBlock, compareTo, copy, copy, copy, dropLastRowsOrColumns, equals, estimateSizeDenseInMemory, estimateSizeInMemory, estimateSizeInMemory, estimateSizeOnDisk, estimateSizeOnDisk, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSparsityOnAggBinary, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatOnDisk, evalSparseFormatOnDisk, examSparsity, examSparsity, getDenseBlock, getExactSerializedSize, getInMemorySize, getNonZeros, getNumColumns, getNumRows, getSparseBlock, getSparseBlockIterator, getSparseBlockIterator, getValue, getValueDenseUnsafe, getValueSparseUnsafe, hashCode, init, init, interQuartileMean, interQuartileMeanOLD, isAllocated, isDiag, isEmpty, isEmptyBlock, isInSparseFormat, isShallowSerialize, isThreadSafe, isThreadSafe, isUltraSparse, isVector, leftIndexingOperations, leftIndexingOperations, leftIndexingOperations, max, median, merge, merge, min, minNonZero, pickValue, pickValue, pickValues, print, quickSetValue, randOperations, randOperations, randOperations, randOperations, recomputeNonZeros, recomputeNonZeros, reset, reset, reset, reset, reset, reset, sampleOperations, seqOperations, setDiag, setNonZeros, setNumColumns, setNumRows, setValue, setValueDenseUnsafe, sliceOperations, sortSparseRows, sortSparseRows, sparseToDense, sparseToDense, sum, sumSq, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TRANSPOSE_INPUT

public static final boolean TRANSPOSE_INPUT

See Also:

Constant Field Values

MATERIALIZE_ZEROS

public static final boolean MATERIALIZE_ZEROS

See Also:

Constant Field Values

MIN_PAR_AGG_THRESHOLD

public static final long MIN_PAR_AGG_THRESHOLD

See Also:

Constant Field Values

INVESTIGATE_ESTIMATES

public static boolean INVESTIGATE_ESTIMATES

ALLOW_DDC_ENCODING

public static boolean ALLOW_DDC_ENCODING

_colGroups

protected ArrayList<ColGroup> _colGroups

_stats

protected CompressedMatrixBlock.CompressionStatistics _stats

Constructor Detail

CompressedMatrixBlock

public CompressedMatrixBlock()

CompressedMatrixBlock

public CompressedMatrixBlock(int rl,
                             int cl,
                             boolean sparse)

Main constructor for building a block from scratch.

Parameters:

rl - number of rows in the block

cl - number of columns

sparse - true if the UNCOMPRESSED representation of the block should be sparse

CompressedMatrixBlock

public CompressedMatrixBlock(MatrixBlock mb)

"Copy" constructor to populate this compressed block with the uncompressed contents of a conventional block. Does not compress the block.

Parameters:

mb - matrix block

Method Detail

getColGroups

public ArrayList<ColGroup> getColGroups()

Obtain the column groups.

Returns:

the column groups constructed by the compression process.

getNumColGroups

public int getNumColGroups()

isCompressed

public boolean isCompressed()

Obtain whether this block is in compressed form or not.

Returns:

true if this block is in compressed form; false if the block has not yet been compressed

isSingleUncompressedGroup

public boolean isSingleUncompressedGroup()

isEmptyBlock

public boolean isEmptyBlock(boolean safe)

Overrides:

isEmptyBlock in class MatrixBlock

compress

public void compress()
              throws DMLRuntimeException

Compress the contents of this matrix block. After compression, the uncompressed data is discarded. Attempts to update this block after calling this method currently result in INCORRECT BEHAVIOR, something which should be fixed if we move ahead with this compression strategy. +per column sparsity

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

compress

public void compress(int k)
              throws DMLRuntimeException

Compress block.

Parameters:

k - number of threads

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

getCompressionStatistics

public CompressedMatrixBlock.CompressionStatistics getCompressionStatistics()

decompress

public MatrixBlock decompress()
                       throws DMLRuntimeException

Decompress block.

Returns:

a new uncompressed matrix block containing the contents of this block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

decompress

public MatrixBlock decompress(int k)
                       throws DMLRuntimeException

Decompress block.

Parameters:

k - degree of parallelism

Returns:

a new uncompressed matrix block containing the contents of this block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

estimateCompressedSizeInMemory

public long estimateCompressedSizeInMemory()

Obtain an upper bound on the memory used to store the compressed block.

Returns:

an upper bound on the memory used to store this compressed block considering class overhead.

quickGetValue

public double quickGetValue(int r,
                            int c)

Overrides:

quickGetValue in class MatrixBlock

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from class: MatrixBlock

NOTE: The used estimates must be kept consistent with the respective write functions.

Overrides:

getExactSizeOnDisk in class MatrixBlock

Returns:

exact size on disk

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class MatrixBlock

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class MatrixBlock

Throws:

IOException

readExternal

public void readExternal(ObjectInput is)
                  throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.

Specified by:

readExternal in interface Externalizable

Overrides:

readExternal in class MatrixBlock

Parameters:

is - object input

Throws:

IOException - if IOException occurs

writeExternal

public void writeExternal(ObjectOutput os)
                   throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Specified by:

writeExternal in interface Externalizable

Overrides:

writeExternal in class MatrixBlock

Parameters:

os - object output

Throws:

IOException - if IOException occurs

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 boolean inclZeros)

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 int cgl,
                                 int cgu,
                                 boolean inclZeros)

getDenseRowIterator

public Iterator<double[]> getDenseRowIterator(int rl,
                                              int ru)

getSparseRowIterator

public Iterator<SparseRow> getSparseRowIterator(int rl,
                                                int ru)

countNonZerosPerRow

public int[] countNonZerosPerRow(int rl,
                                 int ru)

scalarOperations

public MatrixValue scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator sop,
                                    MatrixValue result)
                             throws DMLRuntimeException

Overrides:

scalarOperations in class MatrixBlock

Throws:

DMLRuntimeException

appendOperations

public MatrixBlock appendOperations(MatrixBlock that,
                                    MatrixBlock ret)
                             throws DMLRuntimeException

Overrides:

appendOperations in class MatrixBlock

Throws:

DMLRuntimeException

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype)
                                      throws DMLRuntimeException

Overrides:

chainMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype,
                                             int k)
                                      throws DMLRuntimeException

Overrides:

chainMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

aggregateBinaryOperations

public MatrixValue aggregateBinaryOperations(MatrixValue mv1,
                                             MatrixValue mv2,
                                             MatrixValue result,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
                                      throws DMLRuntimeException

Overrides:

aggregateBinaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn,
                                            boolean inCP)
                                     throws DMLRuntimeException

Overrides:

aggregateUnaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn)
                                     throws DMLRuntimeException

Overrides:

aggregateUnaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype)
                                              throws DMLRuntimeException

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype,
                                                     int k)
                                              throws DMLRuntimeException

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

hasUncompressedColGroup

public boolean hasUncompressedColGroup()

unaryOperations

public MatrixValue unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op,
                                   MatrixValue result)
                            throws DMLRuntimeException

Overrides:

unaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

unaryOperationsInPlace

public void unaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.UnaryOperator op)
                            throws DMLRuntimeException

Overrides:

unaryOperationsInPlace in class MatrixBlock

Throws:

DMLRuntimeException

binaryOperations

public MatrixValue binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue,
                                    MatrixValue result)
                             throws DMLRuntimeException

Overrides:

binaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

binaryOperationsInPlace

public void binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue)
                             throws DMLRuntimeException

Overrides:

binaryOperationsInPlace in class MatrixBlock

Throws:

DMLRuntimeException

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue correction,
                                 MatrixValue newWithCorrection)
                          throws DMLRuntimeException

Overrides:

incrementalAggregate in class MatrixBlock

Throws:

DMLRuntimeException

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue newWithCorrection)
                          throws DMLRuntimeException

Overrides:

incrementalAggregate in class MatrixBlock

Throws:

DMLRuntimeException

reorgOperations

public MatrixValue reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op,
                                   MatrixValue ret,
                                   int startRow,
                                   int startColumn,
                                   int length)
                            throws DMLRuntimeException

Overrides:

reorgOperations in class MatrixBlock

Throws:

DMLRuntimeException

appendOperations

public MatrixBlock appendOperations(MatrixBlock that,
                                    MatrixBlock ret,
                                    boolean cbind)
                             throws DMLRuntimeException

Overrides:

appendOperations in class MatrixBlock

Throws:

DMLRuntimeException

appendOperations

public void appendOperations(MatrixValue v2,
                             ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                             int blockRowFactor,
                             int blockColFactor,
                             boolean cbind,
                             boolean m2IsLast,
                             int nextNCol)
                      throws DMLRuntimeException

Overrides:

appendOperations in class MatrixBlock

Throws:

DMLRuntimeException

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val)
                                     throws DMLRuntimeException

Overrides:

permutationMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val,
                                            int k)
                                     throws DMLRuntimeException

Overrides:

permutationMatrixMultOperations in class MatrixBlock

Throws:

DMLRuntimeException

leftIndexingOperations

public MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix,
                                          int rl,
                                          int ru,
                                          int cl,
                                          int cu,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
                                   throws DMLRuntimeException

Overrides:

leftIndexingOperations in class MatrixBlock

Throws:

DMLRuntimeException

leftIndexingOperations

public MatrixBlock leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar,
                                          int rl,
                                          int cl,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
                                   throws DMLRuntimeException

Description copied from class: MatrixBlock

Explicitly allow left indexing for scalars. Note: This operation is now 0-based. * Operations to be performed: 1) result=this; 2) result[row,column] = scalar.getDoubleValue();

Overrides:

leftIndexingOperations in class MatrixBlock

Parameters:

scalar - scalar object

rl - row lower

cl - column lower

ret - ?

update - ?

Returns:

matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

sliceOperations

public MatrixBlock sliceOperations(int rl,
                                   int ru,
                                   int cl,
                                   int cu,
                                   org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret)
                            throws DMLRuntimeException

Description copied from class: MatrixBlock

Method to perform rangeReIndex operation for a given lower and upper bounds in row and column dimensions. Extracted submatrix is returned as "result". Note: This operation is now 0-based.

Specified by:

sliceOperations in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Overrides:

sliceOperations in class MatrixBlock

Parameters:

rl - row lower

ru - row upper

cl - column lower

cu - column upper

ret - ?

Returns:

matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

sliceOperations

public void sliceOperations(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                            org.apache.sysml.runtime.util.IndexRange range,
                            int rowCut,
                            int colCut,
                            int normalBlockRowFactor,
                            int normalBlockColFactor,
                            int boundaryRlen,
                            int boundaryClen)

Overrides:

sliceOperations in class MatrixBlock

zeroOutOperations

public MatrixValue zeroOutOperations(MatrixValue result,
                                     org.apache.sysml.runtime.util.IndexRange range,
                                     boolean complementary)
                              throws DMLRuntimeException

Overrides:

zeroOutOperations in class MatrixBlock

Throws:

DMLRuntimeException

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)
                                                                    throws DMLRuntimeException

Overrides:

cmOperations in class MatrixBlock

Throws:

DMLRuntimeException

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op,
                                                                           MatrixBlock weights)
                                                                    throws DMLRuntimeException

Overrides:

cmOperations in class MatrixBlock

Throws:

DMLRuntimeException

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that)
                                                                     throws DMLRuntimeException

Overrides:

covOperations in class MatrixBlock

Throws:

DMLRuntimeException

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that,
                                                                            MatrixBlock weights)
                                                                     throws DMLRuntimeException

Overrides:

covOperations in class MatrixBlock

Throws:

DMLRuntimeException

sortOperations

public MatrixValue sortOperations(MatrixValue weights,
                                  MatrixValue result)
                           throws DMLRuntimeException

Overrides:

sortOperations in class MatrixBlock

Throws:

DMLRuntimeException

aggregateBinaryOperations

public MatrixValue aggregateBinaryOperations(MatrixIndexes m1Index,
                                             MatrixValue m1Value,
                                             MatrixIndexes m2Index,
                                             MatrixValue m2Value,
                                             MatrixValue result,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
                                      throws DMLRuntimeException

Overrides:

aggregateBinaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

aggregateTernaryOperations

public MatrixBlock aggregateTernaryOperations(MatrixBlock m1,
                                              MatrixBlock m2,
                                              MatrixBlock m3,
                                              MatrixBlock ret,
                                              org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op,
                                              boolean inCP)
                                       throws DMLRuntimeException

Overrides:

aggregateTernaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

uaggouterchainOperations

public MatrixBlock uaggouterchainOperations(MatrixBlock mbLeft,
                                            MatrixBlock mbRight,
                                            MatrixBlock mbOut,
                                            org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp,
                                            org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)
                                     throws DMLRuntimeException

Overrides:

uaggouterchainOperations in class MatrixBlock

Throws:

DMLRuntimeException

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op)
                                 throws DMLRuntimeException

Description copied from class: MatrixBlock

Invocation from CP instructions. The aggregate is computed on the groups object against target and weights. Notes: * The computed number of groups is reused for multiple invocations with different target. * This implementation supports that the target is passed as column or row vector, in case of row vectors we also use sparse-safe implementations for sparse safe aggregation operators.

Overrides:

groupedAggOperations in class MatrixBlock

Parameters:

tgt - ?

wghts - ?

ret - ?

ngroups - ?

op - operator

Returns:

matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op,
                                        int k)
                                 throws DMLRuntimeException

Overrides:

groupedAggOperations in class MatrixBlock

Throws:

DMLRuntimeException

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         MatrixBlock select)
                                  throws DMLRuntimeException

Overrides:

removeEmptyOperations in class MatrixBlock

Throws:

DMLRuntimeException

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows)
                                  throws DMLRuntimeException

Overrides:

removeEmptyOperations in class MatrixBlock

Throws:

DMLRuntimeException

rexpandOperations

public MatrixBlock rexpandOperations(MatrixBlock ret,
                                     double max,
                                     boolean rows,
                                     boolean cast,
                                     boolean ignore,
                                     int k)
                              throws DMLRuntimeException

Overrides:

rexpandOperations in class MatrixBlock

Throws:

DMLRuntimeException

replaceOperations

public MatrixValue replaceOperations(MatrixValue result,
                                     double pattern,
                                     double replacement)
                              throws DMLRuntimeException

Overrides:

replaceOperations in class MatrixBlock

Throws:

DMLRuntimeException

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              double scalar,
                              MatrixValue that,
                              CTableMap resultMap,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)

Overrides:

ternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              double scalar,
                              double scalar2,
                              CTableMap resultMap,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)

Overrides:

ternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              MatrixIndexes ix1,
                              double scalar,
                              boolean left,
                              int brlen,
                              CTableMap resultMap,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

Specific ctable case of ctable(seq(...),X), where X is the only matrix input. The 'left' input parameter specifies if the seq appeared on the left, otherwise it appeared on the right.

Overrides:

ternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              MatrixValue that,
                              double scalar,
                              boolean ignoreZeros,
                              CTableMap resultMap,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices. In case of matrices and ignoreZeros=true we can also use a sparse-safe implementation

Overrides:

ternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              MatrixValue that,
                              double scalar,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)

Overrides:

ternaryOperations in class MatrixBlock

Parameters:

op - operator

that - matrix value

scalar - scalar double

resultBlock - result matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              MatrixValue that,
                              MatrixValue that2,
                              CTableMap resultMap)
                       throws DMLRuntimeException

Description copied from class: MatrixBlock

D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)

Overrides:

ternaryOperations in class MatrixBlock

Parameters:

op - operator

that - matrix value 1

that2 - matrix value 2

resultMap - table map

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

ternaryOperations

public void ternaryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                              MatrixValue that,
                              MatrixValue that2,
                              CTableMap resultMap,
                              MatrixBlock resultBlock)
                       throws DMLRuntimeException

Overrides:

ternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

quaternaryOperations

public MatrixValue quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixValue um,
                                        MatrixValue vm,
                                        MatrixValue wm,
                                        MatrixValue out)
                                 throws DMLRuntimeException

Overrides:

quaternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

quaternaryOperations

public MatrixValue quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixValue um,
                                        MatrixValue vm,
                                        MatrixValue wm,
                                        MatrixValue out,
                                        int k)
                                 throws DMLRuntimeException

Overrides:

quaternaryOperations in class MatrixBlock

Throws:

DMLRuntimeException

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         LongStream nnzInBlock,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed)
                                  throws DMLRuntimeException

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

nnzInBlock - number of nonzeros in block

bigrand - ?

bSeed - seed value

Returns:

matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         LongStream nnzInBlock,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed,
                                         int k)
                                  throws DMLRuntimeException

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

nnzInBlock - number of nonzeros in block

bigrand - ?

bSeed - seed value

k - ?

Returns:

matrix block

Throws:

DMLRuntimeException - if DMLRuntimeException occurs

seqOperationsInPlace

public MatrixBlock seqOperationsInPlace(double from,
                                        double to,
                                        double incr)
                                 throws DMLRuntimeException

Overrides:

seqOperationsInPlace in class MatrixBlock

Throws:

DMLRuntimeException