org.apache.sysml.runtime.compress

Class CompressedMatrixBlock

java.lang.Object

org.apache.sysml.runtime.matrix.data.MatrixValue

org.apache.sysml.runtime.matrix.data.MatrixBlock

org.apache.sysml.runtime.compress.CompressedMatrixBlock

All Implemented Interfaces:

Externalizable, Serializable, Comparable, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, org.apache.sysml.runtime.controlprogram.caching.CacheBlock

public class CompressedMatrixBlock
extends MatrixBlock
implements Externalizable

Experimental version of MatrixBlock that allows a compressed internal representation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompressedMatrixBlock.CompressionStatistics

Nested classes/interfaces inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

MatrixBlock.BlockType, MatrixBlock.SparsityEstimate

Nested classes/interfaces inherited from class org.apache.sysml.runtime.matrix.data.MatrixValue

MatrixValue.CellIndex

Field Summary

Fields

Modifier and Type	Field and Description
protected ArrayList<ColGroup>	_colGroups
protected boolean	_sharedDDC1Dict
protected CompressedMatrixBlock.CompressionStatistics	_stats
static boolean	ALLOW_DDC_ENCODING
static boolean	ALLOW_SHARED_DDC1_DICTIONARY
static boolean	INVESTIGATE_ESTIMATES
static boolean	MATERIALIZE_ZEROS
static long	MIN_PAR_AGG_THRESHOLD
static boolean	TRANSPOSE_INPUT

Fields inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

clen, CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE, DEFAULT_INPLACE_SPARSEBLOCK, DEFAULT_SPARSEBLOCK, denseBlock, estimatedNNzsPerRow, HEADER_SIZE, MAX_SHALLOW_SERIALIZE_OVERHEAD, nonZeros, rlen, sparse, sparseBlock, SPARSITY_TURN_POINT, ULTRA_SPARSE_BLOCK_NNZ, ULTRA_SPARSITY_TURN_POINT, ULTRA_SPARSITY_TURN_POINT2

Constructor Summary

Constructors

Constructor and Description
CompressedMatrixBlock()
CompressedMatrixBlock(int rl, int cl, boolean sparse) Main constructor for building a block from scratch.
CompressedMatrixBlock(MatrixBlock mb) "Copy" constructor to populate this compressed block with the uncompressed contents of a conventional block.

Method Summary

Modifier and Type	Method and Description
MatrixBlock	aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixBlock	aggregateBinaryOperations(MatrixIndexes m1Index, MatrixBlock m1Value, MatrixIndexes m2Index, MatrixBlock m2Value, MatrixBlock result, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixBlock	aggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op, boolean inCP)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn, boolean inCP)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret, boolean cbind)
void	append(MatrixValue v2, ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, int blockRowFactor, int blockColFactor, boolean cbind, boolean m2IsLast, int nextNCol)
MatrixValue	binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue, MatrixValue result)
void	binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype, int k)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op, MatrixBlock weights)
MatrixBlock	compress() Compress the contents of this matrix block.
MatrixBlock	compress(int k) Compress block.
int[]	countNonZerosPerRow(int rl, int ru)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that, MatrixBlock weights)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalar, double scalar2, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalar, MatrixValue that, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixIndexes ix1, double scalar, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock) Specific ctable case of ctable(seq(...),X), where X is the only matrix input.
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, double scalar, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices.
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap) D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue that, MatrixValue that2, CTableMap resultMap, MatrixBlock resultBlock)
MatrixBlock	ctableSeqOperations(MatrixValue that, double scalar, MatrixBlock resultBlock) D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)
MatrixBlock	decompress() Decompress block.
MatrixBlock	decompress(int k) Decompress block.
long	estimateCompressedSizeInMemory() Obtain an upper bound on the memory used to store the compressed block.
ArrayList<ColGroup>	getColGroups() Obtain the column groups.
CompressedMatrixBlock.CompressionStatistics	getCompressionStatistics()
Iterator<double[]>	getDenseRowIterator(int rl, int ru)
long	getExactSizeOnDisk() NOTE: The used estimates must be kept consistent with the respective write functions.
Iterator<IJV>	getIterator(int rl, int ru, boolean inclZeros)
Iterator<IJV>	getIterator(int rl, int ru, int cgl, int cgu, boolean inclZeros)
int	getNumColGroups()
Iterator<SparseRow>	getSparseRowIterator(int rl, int ru)
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op) Invocation from CP instructions.
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op, int k)
boolean	hasUncompressedColGroup()
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue newWithCorrection)
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection, boolean deep)
boolean	isCompressed() Obtain whether this block is in compressed form or not.
boolean	isEmptyBlock(boolean safe)
boolean	isShallowSerialize() Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isShallowSerialize(boolean inclConvert) Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isSingleUncompressedGroup()
MatrixBlock	leftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
MatrixBlock	leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar, int rl, int cl, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update) Explicitly allow left indexing for scalars.
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val)
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)
MatrixBlock	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out)
MatrixBlock	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out, int k)
double	quickGetValue(int r, int c)
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed) Function to generate a matrix of random numbers.
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k) Function to generate a matrix of random numbers.
void	readExternal(ObjectInput is) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.
void	readFields(DataInput in)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
MatrixValue	reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)
MatrixValue	replaceOperations(MatrixValue result, double pattern, double replacement)
MatrixBlock	rexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
MatrixValue	scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator sop, MatrixValue result)
MatrixBlock	seqOperationsInPlace(double from, double to, double incr)
void	slice(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, org.apache.sysml.runtime.util.IndexRange range, int rowCut, int colCut, int normalBlockRowFactor, int normalBlockColFactor, int boundaryRlen, int boundaryClen)
MatrixBlock	slice(int rl, int ru, int cl, int cu, org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret) Slice a sub block out of the current block and write into the given output block.
MatrixValue	sortOperations(MatrixValue weights, MatrixValue result)
MatrixBlock	ternaryOperations(org.apache.sysml.runtime.matrix.operators.TernaryOperator op, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret)
void	toShallowSerializeBlock() Converts a cache block that is not shallow serializable into a form that is shallow serializable.
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype, int k)
MatrixBlock	uaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp, org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)
MatrixValue	unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op, MatrixValue result)
void	write(DataOutput out)
void	writeExternal(ObjectOutput os) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.
MatrixValue	zeroOutOperations(MatrixValue result, org.apache.sysml.runtime.util.IndexRange range, boolean complementary)

Methods inherited from class org.apache.sysml.runtime.matrix.data.MatrixBlock

allocateAndResetSparseBlock, allocateBlock, allocateDenseBlock, allocateDenseBlock, allocateDenseBlockUnsafe, allocateSparseRowsBlock, allocateSparseRowsBlock, append, appendRow, appendRow, appendRowToSparse, appendToSparse, appendToSparse, appendValue, checkNaN, checkNonZeros, checkSparseRows, checkSparseRows, cleanupBlock, compactEmptyBlock, compareTo, computeIQMCorrection, copy, copy, copy, copyShallow, dropLastRowsOrColumns, equals, estimateSizeDenseInMemory, estimateSizeInMemory, estimateSizeInMemory, estimateSizeOnDisk, estimateSizeOnDisk, estimateSizeSparseInMemory, estimateSizeSparseInMemory, estimateSparsityOnAggBinary, evalSparseFormatInMemory, evalSparseFormatInMemory, evalSparseFormatOnDisk, evalSparseFormatOnDisk, examSparsity, examSparsity, extractTriangular, getDenseBlock, getDenseBlockValues, getExactSerializedSize, getInMemorySize, getLength, getMatrixCharacteristics, getNonZeros, getNumColumns, getNumRows, getSparseBlock, getSparseBlockIterator, getSparseBlockIterator, getSparsity, getValue, getValueDenseUnsafe, hashCode, init, init, interQuartileMean, isAllocated, isEmpty, isEmptyBlock, isInSparseFormat, isThreadSafe, isThreadSafe, isUltraSparse, isUltraSparse, isUltraSparsePermutationMatrix, isVector, leftIndexingOperations, max, mean, median, merge, merge, merge, min, minNonZero, naryOperations, pickValue, pickValue, pickValues, prod, quickSetValue, randOperations, randOperations, randOperations, randOperations, recomputeNonZeros, recomputeNonZeros, recomputeNonZeros, reset, reset, reset, reset, reset, reset, sampleOperations, seqOperations, setNonZeros, setNumColumns, setNumRows, setSparseBlock, setValue, slice, slice, slice, sortSparseRows, sortSparseRows, sparseToDense, sum, sumSq, sumWeightForQuantile, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TRANSPOSE_INPUT

public static final boolean TRANSPOSE_INPUT

See Also:

Constant Field Values

MATERIALIZE_ZEROS

public static final boolean MATERIALIZE_ZEROS

See Also:

Constant Field Values

MIN_PAR_AGG_THRESHOLD

public static final long MIN_PAR_AGG_THRESHOLD

See Also:

Constant Field Values

INVESTIGATE_ESTIMATES

public static final boolean INVESTIGATE_ESTIMATES

See Also:

Constant Field Values

ALLOW_DDC_ENCODING

public static boolean ALLOW_DDC_ENCODING

ALLOW_SHARED_DDC1_DICTIONARY

public static final boolean ALLOW_SHARED_DDC1_DICTIONARY

See Also:

Constant Field Values

_colGroups

protected ArrayList<ColGroup> _colGroups

_stats

protected CompressedMatrixBlock.CompressionStatistics _stats

_sharedDDC1Dict

protected boolean _sharedDDC1Dict

Constructor Detail

CompressedMatrixBlock

public CompressedMatrixBlock()

CompressedMatrixBlock

public CompressedMatrixBlock(int rl,
                             int cl,
                             boolean sparse)

Main constructor for building a block from scratch.

Parameters:

rl - number of rows in the block

cl - number of columns

sparse - true if the UNCOMPRESSED representation of the block should be sparse

CompressedMatrixBlock

public CompressedMatrixBlock(MatrixBlock mb)

"Copy" constructor to populate this compressed block with the uncompressed contents of a conventional block. Does not compress the block.

Parameters:

mb - matrix block

Method Detail

getColGroups

public ArrayList<ColGroup> getColGroups()

Obtain the column groups.

Returns:

the column groups constructed by the compression process.

getNumColGroups

public int getNumColGroups()

isCompressed

public boolean isCompressed()

Obtain whether this block is in compressed form or not.

Returns:

true if this block is in compressed form; false if the block has not yet been compressed

isSingleUncompressedGroup

public boolean isSingleUncompressedGroup()

isEmptyBlock

public boolean isEmptyBlock(boolean safe)

Overrides:

isEmptyBlock in class MatrixBlock

compress

public MatrixBlock compress()

Compress the contents of this matrix block. After compression, the uncompressed data is discarded. Attempts to update this block after calling this method currently result in INCORRECT BEHAVIOR, something which should be fixed if we move ahead with this compression strategy. +per column sparsity

Returns:

compressed matrix block or original block if incompressible

compress

public MatrixBlock compress(int k)

Compress block.

Parameters:

k - number of threads

Returns:

compressed matrix block or original block if incompressible

getCompressionStatistics

public CompressedMatrixBlock.CompressionStatistics getCompressionStatistics()

decompress

public MatrixBlock decompress()

Decompress block.

Returns:

a new uncompressed matrix block containing the contents of this block

decompress

public MatrixBlock decompress(int k)

Decompress block.

Parameters:

k - degree of parallelism

Returns:

a new uncompressed matrix block containing the contents of this block

estimateCompressedSizeInMemory

public long estimateCompressedSizeInMemory()

Obtain an upper bound on the memory used to store the compressed block.

Returns:

an upper bound on the memory used to store this compressed block considering class overhead.

quickGetValue

public double quickGetValue(int r,
                            int c)

Overrides:

quickGetValue in class MatrixBlock

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from class: MatrixBlock

NOTE: The used estimates must be kept consistent with the respective write functions.

Overrides:

getExactSizeOnDisk in class MatrixBlock

Returns:

exact size on disk

isShallowSerialize

public boolean isShallowSerialize()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Overrides:

isShallowSerialize in class MatrixBlock

Returns:

true if shallow serialized

isShallowSerialize

public boolean isShallowSerialize(boolean inclConvert)

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Overrides:

isShallowSerialize in class MatrixBlock

Parameters:

inclConvert - if true report blocks as shallow serialize that are currently not amenable but can be brought into an amenable form via toShallowSerializeBlock.

Returns:

true if shallow serialized

toShallowSerializeBlock

public void toShallowSerializeBlock()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Converts a cache block that is not shallow serializable into a form that is shallow serializable. This methods has no affect if the given cache block is not amenable.

Specified by:

toShallowSerializeBlock in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Overrides:

toShallowSerializeBlock in class MatrixBlock

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class MatrixBlock

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class MatrixBlock

Throws:

IOException

readExternal

public void readExternal(ObjectInput is)
                  throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.

Specified by:

readExternal in interface Externalizable

Overrides:

readExternal in class MatrixBlock

Parameters:

is - object input

Throws:

IOException - if IOException occurs

writeExternal

public void writeExternal(ObjectOutput os)
                   throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Specified by:

writeExternal in interface Externalizable

Overrides:

writeExternal in class MatrixBlock

Parameters:

os - object output

Throws:

IOException - if IOException occurs

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 boolean inclZeros)

getIterator

public Iterator<IJV> getIterator(int rl,
                                 int ru,
                                 int cgl,
                                 int cgu,
                                 boolean inclZeros)

getDenseRowIterator

public Iterator<double[]> getDenseRowIterator(int rl,
                                              int ru)

getSparseRowIterator

public Iterator<SparseRow> getSparseRowIterator(int rl,
                                                int ru)

countNonZerosPerRow

public int[] countNonZerosPerRow(int rl,
                                 int ru)

scalarOperations

public MatrixValue scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator sop,
                                    MatrixValue result)

Overrides:

scalarOperations in class MatrixBlock

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret)

Overrides:

append in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype)

Overrides:

chainMatrixMultOperations in class MatrixBlock

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype,
                                             int k)

Overrides:

chainMatrixMultOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixBlock m1,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)

Overrides:

aggregateBinaryOperations in class MatrixBlock

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn,
                                            boolean inCP)

Overrides:

aggregateUnaryOperations in class MatrixBlock

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn)

Overrides:

aggregateUnaryOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype,
                                                     int k)

Overrides:

transposeSelfMatrixMultOperations in class MatrixBlock

hasUncompressedColGroup

public boolean hasUncompressedColGroup()

unaryOperations

public MatrixValue unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op,
                                   MatrixValue result)

Overrides:

unaryOperations in class MatrixBlock

binaryOperations

public MatrixValue binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue,
                                    MatrixValue result)

Overrides:

binaryOperations in class MatrixBlock

binaryOperationsInPlace

public void binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue)

Overrides:

binaryOperationsInPlace in class MatrixBlock

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue correction,
                                 MatrixValue newWithCorrection,
                                 boolean deep)

Overrides:

incrementalAggregate in class MatrixBlock

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue newWithCorrection)

Overrides:

incrementalAggregate in class MatrixBlock

reorgOperations

public MatrixValue reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op,
                                   MatrixValue ret,
                                   int startRow,
                                   int startColumn,
                                   int length)

Overrides:

reorgOperations in class MatrixBlock

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret,
                          boolean cbind)

Overrides:

append in class MatrixBlock

append

public void append(MatrixValue v2,
                   ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                   int blockRowFactor,
                   int blockColFactor,
                   boolean cbind,
                   boolean m2IsLast,
                   int nextNCol)

Overrides:

append in class MatrixBlock

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val)

Overrides:

permutationMatrixMultOperations in class MatrixBlock

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val,
                                            int k)

Overrides:

permutationMatrixMultOperations in class MatrixBlock

leftIndexingOperations

public MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix,
                                          int rl,
                                          int ru,
                                          int cl,
                                          int cu,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)

Overrides:

leftIndexingOperations in class MatrixBlock

leftIndexingOperations

public MatrixBlock leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar,
                                          int rl,
                                          int cl,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)

Description copied from class: MatrixBlock

Explicitly allow left indexing for scalars. Note: This operation is now 0-based. * Operations to be performed: 1) result=this; 2) result[row,column] = scalar.getDoubleValue();

Overrides:

leftIndexingOperations in class MatrixBlock

Parameters:

scalar - scalar object

rl - row lower

cl - column lower

ret - ?

update - ?

Returns:

matrix block

slice

public MatrixBlock slice(int rl,
                         int ru,
                         int cl,
                         int cu,
                         org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret)

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Slice a sub block out of the current block and write into the given output block. This method returns the passed instance if not null.

Specified by:

slice in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Overrides:

slice in class MatrixBlock

Parameters:

rl - row lower

ru - row upper

cl - column lower

cu - column upper

ret - cache block

Returns:

sub-block of cache block

slice

public void slice(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                  org.apache.sysml.runtime.util.IndexRange range,
                  int rowCut,
                  int colCut,
                  int normalBlockRowFactor,
                  int normalBlockColFactor,
                  int boundaryRlen,
                  int boundaryClen)

Overrides:

slice in class MatrixBlock

zeroOutOperations

public MatrixValue zeroOutOperations(MatrixValue result,
                                     org.apache.sysml.runtime.util.IndexRange range,
                                     boolean complementary)

Overrides:

zeroOutOperations in class MatrixBlock

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)

Overrides:

cmOperations in class MatrixBlock

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op,
                                                                           MatrixBlock weights)

Overrides:

cmOperations in class MatrixBlock

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that)

Overrides:

covOperations in class MatrixBlock

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that,
                                                                            MatrixBlock weights)

Overrides:

covOperations in class MatrixBlock

sortOperations

public MatrixValue sortOperations(MatrixValue weights,
                                  MatrixValue result)

Overrides:

sortOperations in class MatrixBlock

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixIndexes m1Index,
                                             MatrixBlock m1Value,
                                             MatrixIndexes m2Index,
                                             MatrixBlock m2Value,
                                             MatrixBlock result,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)

Overrides:

aggregateBinaryOperations in class MatrixBlock

aggregateTernaryOperations

public MatrixBlock aggregateTernaryOperations(MatrixBlock m1,
                                              MatrixBlock m2,
                                              MatrixBlock m3,
                                              MatrixBlock ret,
                                              org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op,
                                              boolean inCP)

Overrides:

aggregateTernaryOperations in class MatrixBlock

uaggouterchainOperations

public MatrixBlock uaggouterchainOperations(MatrixBlock mbLeft,
                                            MatrixBlock mbRight,
                                            MatrixBlock mbOut,
                                            org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp,
                                            org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)

Overrides:

uaggouterchainOperations in class MatrixBlock

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op)

Description copied from class: MatrixBlock

Invocation from CP instructions. The aggregate is computed on the groups object against target and weights. Notes: * The computed number of groups is reused for multiple invocations with different target. * This implementation supports that the target is passed as column or row vector, in case of row vectors we also use sparse-safe implementations for sparse safe aggregation operators.

Overrides:

groupedAggOperations in class MatrixBlock

Parameters:

tgt - ?

wghts - ?

ret - ?

ngroups - ?

op - operator

Returns:

matrix block

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op,
                                        int k)

Overrides:

groupedAggOperations in class MatrixBlock

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn,
                                         MatrixBlock select)

Overrides:

removeEmptyOperations in class MatrixBlock

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn)

Overrides:

removeEmptyOperations in class MatrixBlock

rexpandOperations

public MatrixBlock rexpandOperations(MatrixBlock ret,
                                     double max,
                                     boolean rows,
                                     boolean cast,
                                     boolean ignore,
                                     int k)

Overrides:

rexpandOperations in class MatrixBlock

replaceOperations

public MatrixValue replaceOperations(MatrixValue result,
                                     double pattern,
                                     double replacement)

Overrides:

replaceOperations in class MatrixBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             double scalar,
                             MatrixValue that,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             double scalar,
                             double scalar2,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixIndexes ix1,
                             double scalar,
                             boolean left,
                             int brlen,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

Specific ctable case of ctable(seq(...),X), where X is the only matrix input. The 'left' input parameter specifies if the seq appeared on the left, otherwise it appeared on the right.

Overrides:

ctableOperations in class MatrixBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue that,
                             double scalar,
                             boolean ignoreZeros,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices. In case of matrices and ignoreZeros=true we can also use a sparse-safe implementation

Overrides:

ctableOperations in class MatrixBlock

ctableSeqOperations

public MatrixBlock ctableSeqOperations(MatrixValue that,
                                       double scalar,
                                       MatrixBlock resultBlock)

Description copied from class: MatrixBlock

D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)

Overrides:

ctableSeqOperations in class MatrixBlock

Parameters:

that - matrix value

scalar - scalar double

resultBlock - result matrix block

Returns:

resultBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue that,
                             MatrixValue that2,
                             CTableMap resultMap)

Description copied from class: MatrixBlock

D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)

Overrides:

ctableOperations in class MatrixBlock

Parameters:

op - operator

that - matrix value 1

that2 - matrix value 2

resultMap - table map

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue that,
                             MatrixValue that2,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Overrides:

ctableOperations in class MatrixBlock

ternaryOperations

public MatrixBlock ternaryOperations(org.apache.sysml.runtime.matrix.operators.TernaryOperator op,
                                     MatrixBlock m2,
                                     MatrixBlock m3,
                                     MatrixBlock ret)

Overrides:

ternaryOperations in class MatrixBlock

quaternaryOperations

public MatrixBlock quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixBlock um,
                                        MatrixBlock vm,
                                        MatrixBlock wm,
                                        MatrixBlock out)

Overrides:

quaternaryOperations in class MatrixBlock

quaternaryOperations

public MatrixBlock quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixBlock um,
                                        MatrixBlock vm,
                                        MatrixBlock wm,
                                        MatrixBlock out,
                                        int k)

Overrides:

quaternaryOperations in class MatrixBlock

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed)

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

Returns:

matrix block

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed,
                                         int k)

Description copied from class: MatrixBlock

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Overrides:

randOperationsInPlace in class MatrixBlock

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

k - ?

Returns:

matrix block

seqOperationsInPlace

public MatrixBlock seqOperationsInPlace(double from,
                                        double to,
                                        double incr)

Overrides:

seqOperationsInPlace in class MatrixBlock