org.apache.sysds.runtime.compress

Class CompressedMatrixBlock

java.lang.Object

org.apache.sysds.runtime.matrix.data.MatrixValue

org.apache.sysds.runtime.matrix.data.MatrixBlock

org.apache.sysds.runtime.compress.AbstractCompressedMatrixBlock

org.apache.sysds.runtime.compress.CompressedMatrixBlock

All Implemented Interfaces:

Externalizable, Serializable, Comparable, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, CacheBlock

public class CompressedMatrixBlock
extends AbstractCompressedMatrixBlock

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

MatrixBlock.SparsityEstimate

Nested classes/interfaces inherited from class org.apache.sysds.runtime.matrix.data.MatrixValue

MatrixValue.CellIndex

Field Summary

Fields inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE, DEFAULT_INPLACE_SPARSEBLOCK, DEFAULT_SPARSEBLOCK, HEADER_SIZE, MAX_SHALLOW_SERIALIZE_OVERHEAD, SPARSITY_TURN_POINT, ULTRA_SPARSE_BLOCK_NNZ, ULTRA_SPARSITY_TURN_POINT, ULTRA_SPARSITY_TURN_POINT2

Constructor Summary

Constructors

Constructor and Description
CompressedMatrixBlock() Constructor for building an empty Compressed Matrix block object.

Method Summary

Modifier and Type	Method and Description
MatrixBlock	aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, AggregateBinaryOperator op)
MatrixBlock	aggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn)
MatrixBlock	aggregateUnaryOperations(AggregateUnaryOperator op, MatrixValue result, int blen, MatrixIndexes indexesIn, boolean inCP)
void	allocateColGroupList(List<ColGroup> colGroups)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret)
static long	baseSizeInMemory()
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, MapMultChain.ChainType ctype, int k)
int[]	countNonZerosPerRow(int rl, int ru)
MatrixBlock	decompress() Decompress block.
MatrixBlock	decompress(int k) Decompress block.
long	estimateCompressedSizeInMemory() Obtain an upper bound on the memory used to store the compressed block.
List<ColGroup>	getColGroups()
Iterator<double[]>	getDenseRowIterator(int rl, int ru)
long	getExactSizeOnDisk() NOTE: The used estimates must be kept consistent with the respective write functions.
Iterator<IJV>	getIterator(int rl, int ru, boolean inclZeros)
Iterator<IJV>	getIterator(int rl, int ru, int cgl, int cgu, boolean inclZeros)
Iterator<SparseRow>	getSparseRowIterator(int rl, int ru)
boolean	hasUncompressedColGroup()
boolean	isSingleUncompressedGroup()
double	quickGetValue(int r, int c)
void	readExternal(ObjectInput is) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.
void	readFields(DataInput in)
MatrixBlock	scalarOperations(ScalarOperator sop, MatrixValue result)
String	toString()
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, MMTSJ.MMTSJType tstype, int k)
void	write(DataOutput out)
void	writeExternal(ObjectOutput os) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Methods inherited from class org.apache.sysds.runtime.compress.AbstractCompressedMatrixBlock

aggregateBinaryOperations, aggregateTernaryOperations, append, append, binaryOperations, binaryOperationsInPlace, cmOperations, cmOperations, covOperations, covOperations, ctableOperations, ctableOperations, ctableOperations, ctableOperations, ctableOperations, ctableOperations, ctableSeqOperations, estimateOriginalSizeInMemory, groupedAggOperations, groupedAggOperations, incrementalAggregate, incrementalAggregate, isEmptyBlock, isShallowSerialize, isShallowSerialize, leftIndexingOperations, leftIndexingOperations, permutationMatrixMultOperations, permutationMatrixMultOperations, quaternaryOperations, quaternaryOperations, randOperationsInPlace, randOperationsInPlace, removeEmptyOperations, removeEmptyOperations, reorgOperations, replaceOperations, rexpandOperations, seqOperationsInPlace, slice, slice, sortOperations, ternaryOperations, toShallowSerializeBlock, uaggouterchainOperations, unaryOperations, zeroOutOperations

Methods inherited from class org.apache.sysds.runtime.matrix.data.MatrixBlock

allocateAndResetSparseBlock, allocateBlock, allocateBlockAsync, allocateDenseBlock, allocateDenseBlock, allocateDenseBlockUnsafe, allocateSparseRowsBlock, allocateSparseRowsBlock, append, appendRow, appendRow, appendRowToSparse, appendToSparse, appendToSparse, appendValue, checkNaN, checkNonZeros, checkSparseRows, checkSparseRows, cleanupBlock, compactEmptyBlock, compareTo, computeIQMCorrection, containsValue, copy, copy, copy, copyShallow, dropLastRowsOrColumns, equals, estimateSizeDenseInMemory, estimateSizeInMemory, estimateSizeInMemory, estimateSizeOnDisk, estimateSizeOnDisk, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSparsityOnAggBinary, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatOnDisk, evalSparseFormatOnDisk, examSparsity, examSparsity, extractTriangular, getDataCharacteristics, getDenseBlock, getDenseBlockValues, getExactSerializedSize, getInMemorySize, getLength, getNonZeros, getNumColumns, getNumRows, getSparseBlock, getSparseBlockIterator, getSparseBlockIterator, getSparsity, getValue, getValueDenseUnsafe, hashCode, init, init, interQuartileMean, isAllocated, isEmpty, isEmptyBlock, isInSparseFormat, isSparsePermutationMatrix, isThreadSafe, isThreadSafe, isUltraSparse, isUltraSparse, isVector, leftIndexingOperations, max, mean, median, merge, merge, merge, merge, min, minNonZero, naryOperations, pickValue, pickValue, pickValues, prod, quickSetValue, randOperations, randOperations, randOperations, randOperations, recomputeNonZeros, recomputeNonZeros, recomputeNonZeros, reset, reset, reset, reset, reset, reset, sampleOperations, seqOperations, setNonZeros, setNumColumns, setNumRows, setSparseBlock, setValue, slice, slice, slice, sortSparseRows, sortSparseRows, sparseToDense, sum, sumSq, sumWeightForQuantile

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

CompressedMatrixBlock

public CompressedMatrixBlock()

Constructor for building an empty Compressed Matrix block object. OBS! Only to be used for serialization.

Method Detail

isSingleUncompressedGroup

public boolean isSingleUncompressedGroup()

allocateColGroupList

public void allocateColGroupList(List<ColGroup> colGroups)

getColGroups

public List<ColGroup> getColGroups()

decompress

public MatrixBlock decompress()

Decompress block.

Specified by:

decompress in class AbstractCompressedMatrixBlock

Returns:

a new uncompressed matrix block containing the contents of this block

decompress

public MatrixBlock decompress(int k)

Decompress block.

Parameters:

k - degree of parallelism

Returns:

a new uncompressed matrix block containing the contents of this block

estimateCompressedSizeInMemory

public long estimateCompressedSizeInMemory()

Obtain an upper bound on the memory used to store the compressed block.

Returns:

an upper bound on the memory used to store this compressed block considering class overhead.

baseSizeInMemory

public static long baseSizeInMemory()

quickGetValue

public double quickGetValue(int r,
                            int c)

Overrides:

quickGetValue in class MatrixBlock

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from class: MatrixBlock

NOTE: The used estimates must be kept consistent with the respective write functions.

Overrides:

getExactSizeOnDisk in class MatrixBlock

Returns:

exact size on disk

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class MatrixBlock

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class MatrixBlock

Throws:

IOException

readExternal

public void readExternal(ObjectInput is)
                  throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd de-serialization.

Specified by:

readExternal in interface Externalizable

Overrides:

readExternal in class MatrixBlock

Parameters:

is - object input

Throws:

IOException - if IOException occurs

writeExternal

public void writeExternal(ObjectOutput os)
                   throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Specified by:

writeExternal in interface Externalizable

Overrides:

writeExternal in class MatrixBlock

Parameters:

os - object output

Throws:

IOException - if IOException occurs

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 boolean inclZeros)

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 int cgl,
                                 int cgu,
                                 boolean inclZeros)

getDenseRowIterator

public Iterator<double[]> getDenseRowIterator(int rl,
                                              int ru)

getSparseRowIterator

public Iterator<SparseRow> getSparseRowIterator(int rl,
                                                int ru)

countNonZerosPerRow

public int[] countNonZerosPerRow(int rl,
                                 int ru)

scalarOperations

public MatrixBlock scalarOperations(ScalarOperator sop,
                                    MatrixValue result)

Overrides:

scalarOperations in class MatrixBlock

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret)

Overrides:

append in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             MapMultChain.ChainType ctype)

Overrides:

chainMatrixMultOperations in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             MapMultChain.ChainType ctype,
                                             int k)

Overrides:

chainMatrixMultOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixBlock m1,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             AggregateBinaryOperator op)

Overrides:

aggregateBinaryOperations in class MatrixBlock

aggregateUnaryOperations

public MatrixBlock aggregateUnaryOperations(AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blen,
                                            MatrixIndexes indexesIn,
                                            boolean inCP)

Overrides:

aggregateUnaryOperations in class MatrixBlock

aggregateUnaryOperations

public MatrixBlock aggregateUnaryOperations(AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blen,
                                            MatrixIndexes indexesIn)

Overrides:

aggregateUnaryOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     MMTSJ.MMTSJType tstype)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     MMTSJ.MMTSJType tstype,
                                                     int k)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

hasUncompressedColGroup

public boolean hasUncompressedColGroup()

toString

public String toString()

Overrides:

toString in class MatrixBlock