org.apache.sysds.runtime.compress

Class CompressedMatrixBlock

java.lang.Object

org.apache.sysds.runtime.matrix.data.MatrixValue

org.apache.sysds.runtime.matrix.data.MatrixBlock

org.apache.sysds.runtime.compress.CompressedMatrixBlock

All Implemented Interfaces:

Externalizable, Serializable, Comparable, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, CacheBlock

public class CompressedMatrixBlock
extends MatrixBlock

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

MatrixBlock.SparsityEstimate

Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixValue

MatrixValue.CellIndex

Field Summary

Fields inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE, DEFAULT_INPLACE_SPARSEBLOCK, DEFAULT_SPARSEBLOCK, HEADER_SIZE, MAX_SHALLOW_SERIALIZE_OVERHEAD, SPARSITY_TURN_POINT, ULTRA_SPARSE_BLOCK_NNZ, ULTRA_SPARSITY_TURN_POINT, ULTRA_SPARSITY_TURN_POINT2

Constructor Summary

Constructors

Constructor and Description
CompressedMatrixBlock() Constructor for building an empty Compressed Matrix block object.
CompressedMatrixBlock(CompressedMatrixBlock that)
CompressedMatrixBlock(int rl, int cl) Main constructor for building a block from scratch.

Method Summary

Modifier and Type	Method and Description
MatrixBlock	aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op)
MatrixBlock	aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op, boolean transposeLeft, boolean transposeRight)
MatrixBlock	aggregateBinaryOperations(MatrixIndexes m1Index, MatrixBlock m1Value, MatrixIndexes m2Index, MatrixBlock m2Value, MatrixBlock result, AggregateBinaryOperator op)
MatrixBlock	aggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, AggregateTernaryOperator op, boolean inCP)
MatrixBlock	aggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn)
MatrixBlock	aggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn, boolean inCP)
void	allocateColGroup(AColGroup cg) Allocate the given column group and remove all references to old column groups.
void	allocateColGroupList(List<AColGroup> colGroups)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret, boolean cbind)
void	append(MatrixValue v2, ArrayList<IndexedMatrixValue> outlist, int blen, boolean cbind, boolean m2IsLast, int nextNCol)
static long	baseSizeInMemory()
MatrixBlock	binaryOperations(BinaryOperator op, MatrixValue thatValue, MatrixValue result)
MatrixBlock	binaryOperationsInPlace(BinaryOperator op, MatrixValue thatValue)
MatrixBlock	binaryOperationsLeft(BinaryOperator op, MatrixValue thatValue, MatrixValue result)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype, int k)
CM_COV_Object	cmOperations(CMOperator op)
CM_COV_Object	cmOperations(CMOperator op, MatrixBlock weights)
boolean	containsValue(double pattern)
void	copy(MatrixValue thatValue) Copy that MatrixValue into this MatrixValue.
void	copy(MatrixValue thatValue, boolean sp) Copy that MatrixValue into this MatrixValue.
CM_COV_Object	covOperations(COVOperator op, MatrixBlock that)
CM_COV_Object	covOperations(COVOperator op, MatrixBlock that, MatrixBlock weights)
void	ctableOperations(Operator op, double scalar, double scalar2, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)
void	ctableOperations(Operator op, double scalar, MatrixValue that, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)
void	ctableOperations(Operator op, MatrixIndexes ix1, double scalar, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock) Specific ctable case of ctable(seq(...),X), where X is the only matrix input.
void	ctableOperations(Operator op, MatrixValue that, double scalar, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices.
void	ctableOperations(Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap) D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)
void	ctableOperations(Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap, MatrixBlock resultBlock)
MatrixBlock	ctableSeqOperations(MatrixValue that, double scalar, MatrixBlock resultBlock) D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)
MatrixBlock	decompress() Decompress block.
MatrixBlock	decompress(int k) Decompress block.
MatrixBlock	decompress(MatrixBlock ret, int k)
long	estimateCompressedSizeInMemory() Obtain an upper bound on the memory used to store the compressed block.
List<AColGroup>	getColGroups()
long	getExactSizeOnDisk() NOTE: The used estimates must be kept consistent with the respective write functions.
org.apache.commons.lang3.tuple.Pair<Integer,int[]>	getMaxNumValues()
MatrixBlock	getUncompressed()
static MatrixBlock	getUncompressed(MatrixValue mVal)
ColGroupUncompressed	getUncompressedColGroup()
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, Operator op) Invocation from CP instructions.
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, Operator op, int k)
void	incrementalAggregate(AggregateOperator aggOp, MatrixValue newWithCorrection)
void	incrementalAggregate(AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection, boolean deep)
boolean	isEmptyBlock(boolean safe)
boolean	isOverlapping()
boolean	isShallowSerialize() Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isShallowSerialize(boolean inclConvert) Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isSingleUncompressedGroup()
MatrixBlock	leftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, MatrixObject.UpdateType update)
MatrixBlock	leftIndexingOperations(ScalarObject scalar, int rl, int cl, MatrixBlock ret, MatrixObject.UpdateType update) Explicitly allow left indexing for scalars.
double	max() Wrapper method for reduceall-max of a matrix.
double	min() Wrapper method for reduceall-min of a matrix.
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val)
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)
MatrixBlock	quaternaryOperations(QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out)
MatrixBlock	quaternaryOperations(QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out, int k)
double	quickGetValue(int r, int c)
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed) Function to generate a matrix of random numbers.
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k) Function to generate a matrix of random numbers.
void	readExternal(ObjectInput is) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.
void	readFields(DataInput in)
long	recomputeNonZeros() Recomputes and materializes the number of non-zero values of the entire matrix block.
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
MatrixBlock	reorgOperations(ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)
MatrixBlock	replaceOperations(MatrixValue result, double pattern, double replacement)
MatrixBlock	rexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
MatrixBlock	scalarOperations(ScalarOperator sop, MatrixValue result)
MatrixBlock	seqOperationsInPlace(double from, double to, double incr)
void	setOverlapping(boolean overlapping)
void	slice(ArrayList<IndexedMatrixValue> outlist, IndexRange range, int rowCut, int colCut, int blen, int boundaryRlen, int boundaryClen) Slice out up to 4 matrixBlocks that are separated by the row and col Cuts.
MatrixBlock	slice(int rl, int ru, int cl, int cu, boolean deep, CacheBlock ret) Method to perform rightIndex operation for a given lower and upper bounds in row and column dimensions.
MatrixBlock	sortOperations(MatrixValue weights, MatrixBlock result)
CompressedMatrixBlock	squash(int k)
double	sum() Wrapper method for reduceall-sum of a matrix.
double	sumSq() Wrapper method for reduceall-sumSq of a matrix.
MatrixBlock	ternaryOperations(TernaryOperator op, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret)
void	toShallowSerializeBlock() Converts a cache block that is not shallow serializable into a form that is shallow serializable.
String	toString()
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype, int k)
MatrixBlock	uaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, BinaryOperator bOp, AggregateUnaryOperator uaggOp)
MatrixBlock	unaryOperations(UnaryOperator op, MatrixValue result)
void	write(DataOutput out)
void	writeExternal(ObjectOutput os) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.
MatrixBlock	zeroOutOperations(MatrixValue result, IndexRange range, boolean complementary)

Methods inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

allocateAndResetSparseBlock, allocateBlock, allocateBlockAsync, allocateDenseBlock, allocateDenseBlock, allocateDenseBlockUnsafe, allocateSparseRowsBlock, allocateSparseRowsBlock, append, appendRow, appendRow, appendRowToSparse, appendToSparse, appendToSparse, appendValue, appendValuePlain, checkNaN, checkNonZeros, checkSparseRows, checkSparseRows, cleanupBlock, compactEmptyBlock, compareTo, computeIQMCorrection, copy, copyShallow, ctableSeqOperations, dropLastRowsOrColumns, equals, estimateSizeDenseInMemory, estimateSizeInMemory, estimateSizeInMemory, estimateSizeOnDisk, estimateSizeOnDisk, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSparsityOnAggBinary, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatOnDisk, evalSparseFormatOnDisk, examSparsity, examSparsity, extractTriangular, getDataCharacteristics, getDenseBlock, getDenseBlockValues, getExactSerializedSize, getHeaderSize, getInMemorySize, getLength, getNonZeros, getNumColumns, getNumRows, getSparseBlock, getSparseBlockIterator, getSparseBlockIterator, getSparsity, getValue, getValueDenseUnsafe, hashCode, init, init, interQuartileMean, isAllocated, isEmpty, isEmptyBlock, isInSparseFormat, isSparsePermutationMatrix, isThreadSafe, isThreadSafe, isUltraSparse, isUltraSparse, isVector, leftIndexingOperations, mean, median, merge, merge, merge, merge, minNonZero, naryOperations, pickValue, pickValue, pickValues, prepareAggregateUnaryOutput, prod, quickGetValueThreadSafe, quickSetValue, quickSetValueThreadSafe, randOperations, randOperations, randOperations, randOperations, recomputeNonZeros, recomputeNonZeros, reset, reset, reset, reset, reset, reset, sampleOperations, seqOperations, setDenseBlock, setNonZeros, setNumColumns, setNumRows, setSparseBlock, setValue, slice, slice, slice, slice, slice, slice, sortSparseRows, sortSparseRows, sparseToDense, sumWeightForQuantile

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

CompressedMatrixBlock

public CompressedMatrixBlock()

Constructor for building an empty Compressed Matrix block object. OBS! Only to be used for serialization.

CompressedMatrixBlock

public CompressedMatrixBlock(int rl,
                             int cl)

Main constructor for building a block from scratch. Use with caution, since it constructs an empty matrix block with nothing inside.

Parameters:

rl - number of rows in the block

cl - number of columns

CompressedMatrixBlock

public CompressedMatrixBlock(CompressedMatrixBlock that)

Method Detail

isSingleUncompressedGroup

public boolean isSingleUncompressedGroup()

allocateColGroup

public void allocateColGroup(AColGroup cg)

Allocate the given column group and remove all references to old column groups. This is done by simply allocating a ned _colGroups list and adding the given column group

Parameters:

cg - The column group to use after.

allocateColGroupList

public void allocateColGroupList(List<AColGroup> colGroups)

getColGroups

public List<AColGroup> getColGroups()

decompress

public MatrixBlock decompress()

Decompress block.

Returns:

a new uncompressed matrix block containing the contents of this block

decompress

public MatrixBlock decompress(int k)

Decompress block.

Parameters:

k - degree of parallelism

Returns:

a new uncompressed matrix block containing the contents of this block

decompress

public MatrixBlock decompress(MatrixBlock ret,
                              int k)

squash

public CompressedMatrixBlock squash(int k)

recomputeNonZeros

public long recomputeNonZeros()

Description copied from class: MatrixBlock

Recomputes and materializes the number of non-zero values of the entire matrix block.

Overrides:

recomputeNonZeros in class MatrixBlock

Returns:

number of non-zeros

estimateCompressedSizeInMemory

public long estimateCompressedSizeInMemory()

Obtain an upper bound on the memory used to store the compressed block.

Returns:

an upper bound on the memory used to store this compressed block considering class overhead.

baseSizeInMemory

public static long baseSizeInMemory()

quickGetValue

public double quickGetValue(int r,
                            int c)

Overrides:

quickGetValue in class MatrixBlock

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from class: MatrixBlock

NOTE: The used estimates must be kept consistent with the respective write functions.

Overrides:

getExactSizeOnDisk in class MatrixBlock

Returns:

exact size on disk

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class MatrixBlock

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class MatrixBlock

Throws:

IOException

readExternal

public void readExternal(ObjectInput is)
                  throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.

Specified by:

readExternal in interface Externalizable

Overrides:

readExternal in class MatrixBlock

Parameters:

is - object input

Throws:

IOException - if IOException occurs

writeExternal

public void writeExternal(ObjectOutput os)
                   throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Specified by:

writeExternal in interface Externalizable

Overrides:

writeExternal in class MatrixBlock

Parameters:

os - object output

Throws:

IOException - if IOException occurs

scalarOperations

public MatrixBlock scalarOperations(ScalarOperator sop,
                                    MatrixValue result)

Overrides:

scalarOperations in class MatrixBlock

binaryOperations

public MatrixBlock binaryOperations(BinaryOperator op,
                                    MatrixValue thatValue,
                                    MatrixValue result)

Overrides:

binaryOperations in class MatrixBlock

binaryOperationsLeft

public MatrixBlock binaryOperationsLeft(BinaryOperator op,
                                        MatrixValue thatValue,
                                        MatrixValue result)

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret)

Overrides:

append in class MatrixBlock

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret,
                          boolean cbind)

Overrides:

append in class MatrixBlock

append

public void append(MatrixValue v2,
                   ArrayList<IndexedMatrixValue> outlist,
                   int blen,
                   boolean cbind,
                   boolean m2IsLast,
                   int nextNCol)

Overrides:

append in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             MapMultChain.ChainType ctype)

Overrides:

chainMatrixMultOperations in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             MapMultChain.ChainType ctype,
                                             int k)

Overrides:

chainMatrixMultOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixBlock m1,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             AggregateBinaryOperator op)

Overrides:

aggregateBinaryOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixBlock m1,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             AggregateBinaryOperator op,
                                             boolean transposeLeft,
                                             boolean transposeRight)

aggregateUnaryOperations

public MatrixBlock aggregateUnaryOperations(AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blen,
                                            MatrixIndexes indexesIn)

Overrides:

aggregateUnaryOperations in class MatrixBlock

aggregateUnaryOperations

public MatrixBlock aggregateUnaryOperations(AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blen,
                                            MatrixIndexes indexesIn,
                                            boolean inCP)

Overrides:

aggregateUnaryOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     MMTSJ.MMTSJType tstype)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     MMTSJ.MMTSJType tstype,
                                                     int k)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

replaceOperations

public MatrixBlock replaceOperations(MatrixValue result,
                                     double pattern,
                                     double replacement)

Overrides:

replaceOperations in class MatrixBlock

reorgOperations

public MatrixBlock reorgOperations(ReorgOperator op,
                                   MatrixValue ret,
                                   int startRow,
                                   int startColumn,
                                   int length)

Overrides:

reorgOperations in class MatrixBlock

getUncompressedColGroup

public ColGroupUncompressed getUncompressedColGroup()

getMaxNumValues

public org.apache.commons.lang3.tuple.Pair<Integer,int[]> getMaxNumValues()

toString

public String toString()

Overrides:

toString in class MatrixBlock

isOverlapping

public boolean isOverlapping()

setOverlapping

public void setOverlapping(boolean overlapping)

slice

public MatrixBlock slice(int rl,
                         int ru,
                         int cl,
                         int cu,
                         boolean deep,
                         CacheBlock ret)

Description copied from class: MatrixBlock

Method to perform rightIndex operation for a given lower and upper bounds in row and column dimensions. Extracted submatrix is returned as "result". Note: This operation is now 0-based. This means that if you call with rl == ru then you get 1 row output.

Overrides:

slice in class MatrixBlock

Parameters:

rl - row lower if this value is bellow 0 or above the number of rows contained in the matrix an execption is thrown

ru - row upper if this value is bellow rl or above the number of rows contained in the matrix an exception is thrown

cl - column lower if this value us bellow 0 or above the number of columns contained in the matrix an exception is thrown

cu - column upper if this value us bellow cl or above the number of columns contained in the matrix an exception is thrown

deep - should perform deep copy, this is relelvant in cases where the matrix is in sparse format, or the entire matrix is sliced out

ret - output sliced out matrix block

Returns:

matrix block output matrix block

slice

public void slice(ArrayList<IndexedMatrixValue> outlist,
                  IndexRange range,
                  int rowCut,
                  int colCut,
                  int blen,
                  int boundaryRlen,
                  int boundaryClen)

Description copied from class: MatrixValue

Slice out up to 4 matrixBlocks that are separated by the row and col Cuts. This is used in the context of spark execution to distributed sliced out matrix blocks of correct block size.

Overrides:

slice in class MatrixBlock

Parameters:

outlist - The output matrix blocks that is extracted from the matrix

range - An index range containing overlapping information.

rowCut - The row to cut and split the matrix.

colCut - The column to cut ans split the matrix.

blen - The Block size of the output matrices.

boundaryRlen - The row length of the edge case matrix block, used for the final blocks that does not have enough rows to construct a full block.

boundaryClen - The col length of the edge case matrix block, used for the final blocks that does not have enough cols to construct a full block.

unaryOperations

public MatrixBlock unaryOperations(UnaryOperator op,
                                   MatrixValue result)

Overrides:

unaryOperations in class MatrixBlock

containsValue

public boolean containsValue(double pattern)

Overrides:

containsValue in class MatrixBlock

max

public double max()

Description copied from class: MatrixBlock

Wrapper method for reduceall-max of a matrix.

Overrides:

max in class MatrixBlock

Returns:

the maximum value of all values in the matrix

min

public double min()

Description copied from class: MatrixBlock

Wrapper method for reduceall-min of a matrix.

Overrides:

min in class MatrixBlock

Returns:

the minimum value of all values in the matrix

sum

public double sum()

Description copied from class: MatrixBlock

Wrapper method for reduceall-sum of a matrix.

Overrides:

sum in class MatrixBlock

Returns:

Sum of the values in the matrix.

sumSq

public double sumSq()

Description copied from class: MatrixBlock

Wrapper method for reduceall-sumSq of a matrix.

Overrides:

sumSq in class MatrixBlock

Returns:

Sum of the squared values in the matrix.

rexpandOperations

public MatrixBlock rexpandOperations(MatrixBlock ret,
                                     double max,
                                     boolean rows,
                                     boolean cast,
                                     boolean ignore,
                                     int k)

Overrides:

rexpandOperations in class MatrixBlock

isEmptyBlock

public boolean isEmptyBlock(boolean safe)

Overrides:

isEmptyBlock in class MatrixBlock

binaryOperationsInPlace

public MatrixBlock binaryOperationsInPlace(BinaryOperator op,
                                           MatrixValue thatValue)

Overrides:

binaryOperationsInPlace in class MatrixBlock

incrementalAggregate

public void incrementalAggregate(AggregateOperator aggOp,
                                 MatrixValue correction,
                                 MatrixValue newWithCorrection,
                                 boolean deep)

Overrides:

incrementalAggregate in class MatrixBlock

incrementalAggregate

public void incrementalAggregate(AggregateOperator aggOp,
                                 MatrixValue newWithCorrection)

Overrides:

incrementalAggregate in class MatrixBlock

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val)

Overrides:

permutationMatrixMultOperations in class MatrixBlock

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val,
                                            int k)

Overrides:

permutationMatrixMultOperations in class MatrixBlock

leftIndexingOperations

public MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix,
                                          int rl,
                                          int ru,
                                          int cl,
                                          int cu,
                                          MatrixBlock ret,
                                          MatrixObject.UpdateType update)

Overrides:

leftIndexingOperations in class MatrixBlock

leftIndexingOperations

public MatrixBlock leftIndexingOperations(ScalarObject scalar,
                                          int rl,
                                          int cl,
                                          MatrixBlock ret,
                                          MatrixObject.UpdateType update)

Description copied from class: MatrixBlock

Explicitly allow left indexing for scalars. Note: This operation is now 0-based. * Operations to be performed: 1) result=this; 2) result[row,column] = scalar.getDoubleValue();

Overrides:

leftIndexingOperations in class MatrixBlock

Parameters:

scalar - scalar object

rl - row lower

cl - column lower

ret - ?

update - ?

Returns:

matrix block

zeroOutOperations

public MatrixBlock zeroOutOperations(MatrixValue result,
                                     IndexRange range,
                                     boolean complementary)

Overrides:

zeroOutOperations in class MatrixBlock

cmOperations

public CM_COV_Object cmOperations(CMOperator op)

Overrides:

cmOperations in class MatrixBlock

cmOperations

public CM_COV_Object cmOperations(CMOperator op,
                                  MatrixBlock weights)

Overrides:

cmOperations in class MatrixBlock

covOperations

public CM_COV_Object covOperations(COVOperator op,
                                   MatrixBlock that)

Overrides:

covOperations in class MatrixBlock

covOperations

public CM_COV_Object covOperations(COVOperator op,
                                   MatrixBlock that,
                                   MatrixBlock weights)

Overrides:

covOperations in class MatrixBlock

sortOperations

public MatrixBlock sortOperations(MatrixValue weights,
                                  MatrixBlock result)

Overrides:

sortOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixIndexes m1Index,
                                             MatrixBlock m1Value,
                                             MatrixIndexes m2Index,
                                             MatrixBlock m2Value,
                                             MatrixBlock result,
                                             AggregateBinaryOperator op)

Overrides:

aggregateBinaryOperations in class MatrixBlock

aggregateTernaryOperations

public MatrixBlock aggregateTernaryOperations(MatrixBlock m1,
                                              MatrixBlock m2,
                                              MatrixBlock m3,
                                              MatrixBlock ret,
                                              AggregateTernaryOperator op,
                                              boolean inCP)

Overrides:

aggregateTernaryOperations in class MatrixBlock

uaggouterchainOperations

public MatrixBlock uaggouterchainOperations(MatrixBlock mbLeft,
                                            MatrixBlock mbRight,
                                            MatrixBlock mbOut,
                                            BinaryOperator bOp,
                                            AggregateUnaryOperator uaggOp)

Overrides:

uaggouterchainOperations in class MatrixBlock

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        Operator op)

Description copied from class: MatrixBlock

Invocation from CP instructions. The aggregate is computed on the groups object against target and weights. Notes: * The computed number of groups is reused for multiple invocations with different target. * This implementation supports that the target is passed as column or row vector, in case of row vectors we also use sparse-safe implementations for sparse safe aggregation operators.

Overrides:

groupedAggOperations in class MatrixBlock

Parameters:

tgt - ?

wghts - ?

ret - ?

ngroups - ?

op - operator

Returns:

matrix block

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        Operator op,
                                        int k)

Overrides:

groupedAggOperations in class MatrixBlock

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn,
                                         MatrixBlock select)

Overrides:

removeEmptyOperations in class MatrixBlock

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn)

Overrides:

removeEmptyOperations in class MatrixBlock

ctableOperations

public void ctableOperations(Operator op,
                             double scalar,
                             MatrixValue that,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(Operator op,
                             double scalar,
                             double scalar2,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(Operator op,
                             MatrixIndexes ix1,
                             double scalar,
                             boolean left,
                             int brlen,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

Specific ctable case of ctable(seq(...),X), where X is the only matrix input. The 'left' input parameter specifies if the seq appeared on the left, otherwise it appeared on the right.

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(Operator op,
                             MatrixValue that,
                             double scalar,
                             boolean ignoreZeros,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices. In case of matrices and ignoreZeros=true we can also use a sparse-safe implementation

Overrides:

ctableOperations in class MatrixBlock

ctableSeqOperations

public MatrixBlock ctableSeqOperations(MatrixValue that,
                                       double scalar,
                                       MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)

Overrides:

ctableSeqOperations in class MatrixBlock

Parameters:

that - matrix value

scalar - scalar double

resultBlock - result matrix block

Returns:

resultBlock

ctableOperations

public void ctableOperations(Operator op,
                             MatrixValue that,
                             MatrixValue that2,
                             CTableMap resultMap)

Description copied from class: MatrixBlock

D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)

Overrides:

ctableOperations in class MatrixBlock

Parameters:

op - operator

that - matrix value 1

that2 - matrix value 2

resultMap - table map

ctableOperations

public void ctableOperations(Operator op,
                             MatrixValue that,
                             MatrixValue that2,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Overrides:

ctableOperations in class MatrixBlock

ternaryOperations

public MatrixBlock ternaryOperations(TernaryOperator op,
                                     MatrixBlock m2,
                                     MatrixBlock m3,
                                     MatrixBlock ret)

Overrides:

ternaryOperations in class MatrixBlock

quaternaryOperations

public MatrixBlock quaternaryOperations(QuaternaryOperator qop,
                                        MatrixBlock um,
                                        MatrixBlock vm,
                                        MatrixBlock wm,
                                        MatrixBlock out)

Overrides:

quaternaryOperations in class MatrixBlock

quaternaryOperations

public MatrixBlock quaternaryOperations(QuaternaryOperator qop,
                                        MatrixBlock um,
                                        MatrixBlock vm,
                                        MatrixBlock wm,
                                        MatrixBlock out,
                                        int k)

Overrides:

quaternaryOperations in class MatrixBlock

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed)

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

Returns:

matrix block

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed,
                                         int k)

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

k - ?

Returns:

matrix block

seqOperationsInPlace

public MatrixBlock seqOperationsInPlace(double from,
                                        double to,
                                        double incr)

Overrides:

seqOperationsInPlace in class MatrixBlock

getUncompressed

public static MatrixBlock getUncompressed(MatrixValue mVal)

getUncompressed

public MatrixBlock getUncompressed()

isShallowSerialize

public boolean isShallowSerialize()

Description copied from interface: CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface CacheBlock

Overrides:

isShallowSerialize in class MatrixBlock

Returns:

true if shallow serialized

isShallowSerialize

public boolean isShallowSerialize(boolean inclConvert)

Description copied from interface: CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface CacheBlock

Overrides:

isShallowSerialize in class MatrixBlock

Parameters:

inclConvert - if true report blocks as shallow serialize that are currently not amenable but can be brought into an amenable form via toShallowSerializeBlock.

Returns:

true if shallow serialized

toShallowSerializeBlock

public void toShallowSerializeBlock()

Description copied from interface: CacheBlock

Converts a cache block that is not shallow serializable into a form that is shallow serializable. This methods has no affect if the given cache block is not amenable.

Specified by:

toShallowSerializeBlock in interface CacheBlock

Overrides:

toShallowSerializeBlock in class MatrixBlock

copy

public void copy(MatrixValue thatValue)

Description copied from class: MatrixValue

Copy that MatrixValue into this MatrixValue. If the MatrixValue is a MatrixBlock evaluate the sparsity of the original matrix, and copy into either a sparse or a dense matrix.

Overrides:

copy in class MatrixBlock

Parameters:

thatValue - object to copy the values into.

copy

public void copy(MatrixValue thatValue,
                 boolean sp)

Description copied from class: MatrixValue

Copy that MatrixValue into this MatrixValue. But select sparse destination block depending on boolean parameter.

Overrides:

copy in class MatrixBlock

Parameters:

thatValue - object to copy the values into.

sp - boolean specifying if output should be forced sparse or dense. (only applicable if the 'that' is a MatrixBlock)