org.apache.sysml.runtime.matrix.data

Class MatrixBlock

java.lang.Object

org.apache.sysml.runtime.matrix.data.MatrixValue

org.apache.sysml.runtime.matrix.data.MatrixBlock

All Implemented Interfaces:

Externalizable, Serializable, Comparable, org.apache.hadoop.io.Writable, org.apache.hadoop.io.WritableComparable, org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Direct Known Subclasses:

CompressedMatrixBlock

public class MatrixBlock
extends MatrixValue
implements org.apache.sysml.runtime.controlprogram.caching.CacheBlock, Externalizable

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MatrixBlock.BlockType
static class	MatrixBlock.SparsityEstimate

Nested classes/interfaces inherited from class org.apache.sysml.runtime.matrix.data.MatrixValue

MatrixValue.CellIndex

Field Summary

Fields

Modifier and Type	Field and Description
protected int	clen
static boolean	CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE
static SparseBlock.Type	DEFAULT_INPLACE_SPARSEBLOCK
static SparseBlock.Type	DEFAULT_SPARSEBLOCK
protected DenseBlock	denseBlock
protected int	estimatedNNzsPerRow
static int	HEADER_SIZE
static double	MAX_SHALLOW_SERIALIZE_OVERHEAD
protected long	nonZeros
protected int	rlen
protected boolean	sparse
protected SparseBlock	sparseBlock
static double	SPARSITY_TURN_POINT
static int	ULTRA_SPARSE_BLOCK_NNZ
static double	ULTRA_SPARSITY_TURN_POINT
static double	ULTRA_SPARSITY_TURN_POINT2

Constructor Summary

Constructors

Constructor and Description
MatrixBlock()
MatrixBlock(double val)
MatrixBlock(int rl, int cl, boolean sp)
MatrixBlock(int rl, int cl, boolean sp, long estnnz)
MatrixBlock(int rl, int cl, double val)
MatrixBlock(int rl, int cl, long estnnz)
MatrixBlock(int rl, int cl, long nnz, SparseBlock sblock) Constructs a sparse MatrixBlock with a given instance of a SparseBlock
MatrixBlock(MatrixBlock that)
MatrixBlock(MatrixBlock that, SparseBlock.Type stype, boolean deep)

Method Summary

Modifier and Type	Method and Description
MatrixBlock	aggregateBinaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixBlock	aggregateBinaryOperations(MatrixIndexes m1Index, MatrixBlock m1, MatrixIndexes m2Index, MatrixBlock m2, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
MatrixBlock	aggregateTernaryOperations(MatrixBlock m1, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret, org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op, boolean inCP)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn)
MatrixValue	aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op, MatrixValue result, int blockingFactorRow, int blockingFactorCol, MatrixIndexes indexesIn, boolean inCP)
void	allocateAndResetSparseBlock(boolean clearNNZ, SparseBlock.Type stype)
MatrixBlock	allocateBlock()
MatrixBlock	allocateDenseBlock()
boolean	allocateDenseBlock(boolean clearNNZ)
void	allocateDenseBlockUnsafe(int rl, int cl) This should be called only in the read and write functions for CP This function should be called before calling any setValueDenseUnsafe()
boolean	allocateSparseRowsBlock()
boolean	allocateSparseRowsBlock(boolean clearNNZ)
MatrixBlock	append(MatrixBlock[] that, MatrixBlock ret, boolean cbind)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret)
MatrixBlock	append(MatrixBlock that, MatrixBlock ret, boolean cbind)
void	append(MatrixValue v2, ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, int blockRowFactor, int blockColFactor, boolean cbind, boolean m2IsLast, int nextNCol)
void	appendRow(int r, SparseRow row)
void	appendRow(int r, SparseRow row, boolean deep)
void	appendRowToSparse(SparseBlock dest, MatrixBlock src, int i, int rowoffset, int coloffset, boolean deep)
void	appendToSparse(MatrixBlock that, int rowoffset, int coloffset)
void	appendToSparse(MatrixBlock that, int rowoffset, int coloffset, boolean deep)
void	appendValue(int r, int c, double v) Append value is only used when values are appended at the end of each row for the sparse representation This can only be called, when the caller knows the access pattern of the block
MatrixValue	binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue, MatrixValue result)
void	binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op, MatrixValue thatValue)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype)
MatrixBlock	chainMatrixMultOperations(MatrixBlock v, MatrixBlock w, MatrixBlock out, org.apache.sysml.lops.MapMultChain.ChainType ctype, int k)
void	checkNaN() Checks for existing NaN values in the matrix block.
void	checkNonZeros() Basic debugging primitive to check correctness of nnz.
void	checkSparseRows()
void	checkSparseRows(int rl, int ru) Basic debugging primitive to check sparse block column ordering.
void	cleanupBlock(boolean dense, boolean sparse) Allows to cleanup all previously allocated sparserows or denseblocks.
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op, MatrixBlock weights)
void	compactEmptyBlock() Free unnecessarily allocated empty block.
int	compareTo(Object arg0)
static double	computeIQMCorrection(double sum, double sum_wt, double q25Part, double q25Val, double q75Part, double q75Val)
void	copy(int rl, int ru, int cl, int cu, MatrixBlock src, boolean awareDestNZ) In-place copy of matrix src into the index range of the existing current matrix.
void	copy(MatrixValue thatValue)
void	copy(MatrixValue thatValue, boolean sp)
MatrixBlock	copyShallow(MatrixBlock that)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that)
org.apache.sysml.runtime.instructions.cp.CM_COV_Object	covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op, MatrixBlock that, MatrixBlock weights)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalarThat, double scalarThat2, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, double scalarThat, MatrixValue that2Val, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixIndexes ix1, double scalarThat, boolean left, int brlen, CTableMap resultMap, MatrixBlock resultBlock) Specific ctable case of ctable(seq(...),X), where X is the only matrix input.
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue thatVal, double scalarThat2, boolean ignoreZeros, CTableMap resultMap, MatrixBlock resultBlock) D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices.
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue thatVal, MatrixValue that2Val, CTableMap resultMap) D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)
void	ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixValue thatVal, MatrixValue that2Val, CTableMap resultMap, MatrixBlock resultBlock)
MatrixBlock	ctableSeqOperations(MatrixValue thatMatrix, double thatScalar, MatrixBlock resultBlock) D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)
void	dropLastRowsOrColumns(org.apache.sysml.lops.PartialAggregate.CorrectionLocationType correctionLocation)
boolean	equals(Object arg0)
static long	estimateSizeDenseInMemory(long nrows, long ncols)
long	estimateSizeInMemory()
static long	estimateSizeInMemory(long nrows, long ncols, double sparsity)
long	estimateSizeOnDisk()
static long	estimateSizeOnDisk(long nrows, long ncols, long nnz)
static long	estimateSizeSparseInMemory(long nrows, long ncols, double sparsity)
static long	estimateSizeSparseInMemory(long nrows, long ncols, double sparsity, SparseBlock.Type stype)
static MatrixBlock.SparsityEstimate	estimateSparsityOnAggBinary(MatrixBlock m1, MatrixBlock m2, org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)
boolean	evalSparseFormatInMemory() Evaluates if this matrix block should be in sparse format in memory.
static boolean	evalSparseFormatInMemory(long nrows, long ncols, long nnz) Evaluates if a matrix block with the given characteristics should be in sparse format in memory.
boolean	evalSparseFormatOnDisk() Evaluates if this matrix block should be in sparse format on disk.
static boolean	evalSparseFormatOnDisk(long nrows, long ncols, long nnz) Evaluates if a matrix block with the given characteristics should be in sparse format on disk (or in any other serialized representation).
void	examSparsity()
void	examSparsity(boolean allowCSR) Evaluates if this matrix block should be in sparse format in memory.
MatrixBlock	extractTriangular(MatrixBlock ret, boolean lower, boolean diag, boolean values)
DenseBlock	getDenseBlock()
double[]	getDenseBlockValues()
long	getExactSerializedSize() Get the exact serialized size in bytes of the cache block.
long	getExactSizeOnDisk() NOTE: The used estimates must be kept consistent with the respective write functions.
long	getInMemorySize() Get the in-memory size in bytes of the cache block.
long	getLength()
MatrixCharacteristics	getMatrixCharacteristics()
long	getNonZeros()
int	getNumColumns()
int	getNumRows()
SparseBlock	getSparseBlock()
Iterator<IJV>	getSparseBlockIterator()
Iterator<IJV>	getSparseBlockIterator(int rl, int ru)
double	getSparsity()
double	getValue(int r, int c)
double	getValueDenseUnsafe(int r, int c)
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op) Invocation from CP instructions.
MatrixBlock	groupedAggOperations(MatrixValue tgt, MatrixValue wghts, MatrixValue ret, int ngroups, org.apache.sysml.runtime.matrix.operators.Operator op, int k)
int	hashCode()
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue newWithCorrection)
void	incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp, MatrixValue correction, MatrixValue newWithCorrection, boolean deep)
void	init(double[][] arr, int r, int c) NOTE: This method is designed only for dense representation.
void	init(double[] arr, int r, int c) NOTE: This method is designed only for dense representation.
double	interQuartileMean()
boolean	isAllocated()
boolean	isEmpty()
boolean	isEmptyBlock()
boolean	isEmptyBlock(boolean safe)
boolean	isInSparseFormat() Returns the current representation (true for sparse).
boolean	isShallowSerialize() Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isShallowSerialize(boolean inclConvert) Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.
boolean	isThreadSafe() Indicates if concurrent modifications of disjoint rows are thread-safe.
static boolean	isThreadSafe(boolean sparse) Indicates if concurrent modifications of disjoint rows are thread-safe.
boolean	isUltraSparse()
boolean	isUltraSparse(boolean checkNnz)
boolean	isUltraSparsePermutationMatrix()
boolean	isVector()
MatrixBlock	leftIndexingOperations(MatrixBlock rhsMatrix, org.apache.sysml.runtime.util.IndexRange ixrange, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
MatrixBlock	leftIndexingOperations(MatrixBlock rhsMatrix, int rl, int ru, int cl, int cu, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)
MatrixBlock	leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar, int rl, int cl, MatrixBlock ret, org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update) Explicitly allow left indexing for scalars.
double	max() Wrapper method for reduceall-max of a matrix.
double	mean() Wrapper method for reduceall-mean of a matrix.
double	median()
void	merge(org.apache.sysml.runtime.controlprogram.caching.CacheBlock that, boolean appendOnly) Merge the given block into the current block.
void	merge(MatrixBlock that, boolean appendOnly) Merge disjoint: merges all non-zero values of the given input into the current matrix block.
void	merge(MatrixBlock that, boolean appendOnly, boolean par)
double	min() Wrapper method for reduceall-min of a matrix.
double	minNonZero() Utility function for computing the min non-zero value.
static MatrixBlock	naryOperations(org.apache.sysml.runtime.matrix.operators.Operator op, MatrixBlock[] matrices, org.apache.sysml.runtime.instructions.cp.ScalarObject[] scalars, MatrixBlock ret)
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val)
void	permutationMatrixMultOperations(MatrixValue m2Val, MatrixValue out1Val, MatrixValue out2Val, int k)
double	pickValue(double quantile)
double	pickValue(double quantile, boolean average)
MatrixValue	pickValues(MatrixValue quantiles, MatrixValue ret)
double	prod() Wrapper method for reduceall-product of a matrix.
MatrixBlock	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixBlock um, MatrixBlock vm, MatrixBlock wm, MatrixBlock out)
MatrixBlock	quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop, MatrixBlock U, MatrixBlock V, MatrixBlock wm, MatrixBlock out, int k)
double	quickGetValue(int r, int c)
void	quickSetValue(int r, int c, double v)
static MatrixBlock	randOperations(int rows, int cols, double sparsity, double min, double max, String pdf, long seed) Function to generate the random matrix with specified dimensions (block sizes are not specified).
static MatrixBlock	randOperations(int rows, int cols, double sparsity, double min, double max, String pdf, long seed, int k) Function to generate the random matrix with specified dimensions (block sizes are not specified).
static MatrixBlock	randOperations(RandomMatrixGenerator rgen, long seed) Function to generate the random matrix with specified dimensions and block dimensions.
static MatrixBlock	randOperations(RandomMatrixGenerator rgen, long seed, int k) Function to generate the random matrix with specified dimensions and block dimensions.
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed) Function to generate a matrix of random numbers.
MatrixBlock	randOperationsInPlace(RandomMatrixGenerator rgen, org.apache.commons.math3.random.Well1024a bigrand, long bSeed, int k) Function to generate a matrix of random numbers.
void	readExternal(ObjectInput is) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.
void	readFields(DataInput in)
long	recomputeNonZeros() Recomputes and materializes the number of non-zero values of the entire matrix block.
long	recomputeNonZeros(int rl, int ru)
long	recomputeNonZeros(int rl, int ru, int cl, int cu) Recomputes the number of non-zero values of a specified range of the matrix block.
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn)
MatrixBlock	removeEmptyOperations(MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
MatrixValue	reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op, MatrixValue ret, int startRow, int startColumn, int length)
MatrixValue	replaceOperations(MatrixValue result, double pattern, double replacement)
void	reset()
void	reset(int rl, int cl)
void	reset(int rl, int cl, boolean sp)
void	reset(int rl, int cl, boolean sp, long estnnz)
void	reset(int rl, int cl, double val)
void	reset(int rl, int cl, long estnnz)
MatrixBlock	rexpandOperations(MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
static MatrixBlock	sampleOperations(long range, int size, boolean replace, long seed)
MatrixValue	scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator op, MatrixValue result)
static MatrixBlock	seqOperations(double from, double to, double incr) Method to generate a sequence according to the given parameters.
MatrixBlock	seqOperationsInPlace(double from, double to, double incr)
long	setNonZeros(long nnz)
void	setNumColumns(int c)
void	setNumRows(int r) NOTE: setNumRows() and setNumColumns() are used only in ternaryInstruction (for contingency tables) and pmm for meta corrections.
void	setSparseBlock(SparseBlock sblock)
void	setValue(int r, int c, double v)
void	slice(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist, org.apache.sysml.runtime.util.IndexRange range, int rowCut, int colCut, int normalBlockRowFactor, int normalBlockColFactor, int boundaryRlen, int boundaryClen)
MatrixBlock	slice(org.apache.sysml.runtime.util.IndexRange ixrange, MatrixBlock ret)
MatrixBlock	slice(int rl, int ru)
MatrixBlock	slice(int rl, int ru, int cl, int cu, boolean deep, org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret) Method to perform rightIndex operation for a given lower and upper bounds in row and column dimensions.
MatrixBlock	slice(int rl, int ru, int cl, int cu, org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret) Slice a sub block out of the current block and write into the given output block.
MatrixValue	sortOperations(MatrixValue weights, MatrixValue result)
void	sortSparseRows() Sorts all existing sparse rows by column indexes.
void	sortSparseRows(int rl, int ru) Sorts all existing sparse rows in range [rl,ru) by column indexes.
void	sparseToDense()
double	sum() Wrapper method for reduceall-sum of a matrix.
double	sumSq() Wrapper method for reduceall-sumSq of a matrix.
double	sumWeightForQuantile() In a given two column matrix, the second column denotes weights.
MatrixBlock	ternaryOperations(org.apache.sysml.runtime.matrix.operators.TernaryOperator op, MatrixBlock m2, MatrixBlock m3, MatrixBlock ret)
void	toShallowSerializeBlock() Converts a cache block that is not shallow serializable into a form that is shallow serializable.
String	toString()
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype)
MatrixBlock	transposeSelfMatrixMultOperations(MatrixBlock out, org.apache.sysml.lops.MMTSJ.MMTSJType tstype, int k)
MatrixBlock	uaggouterchainOperations(MatrixBlock mbLeft, MatrixBlock mbRight, MatrixBlock mbOut, org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp, org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)
MatrixValue	unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op, MatrixValue result)
void	write(DataOutput out)
void	writeExternal(ObjectOutput os) Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.
MatrixValue	zeroOutOperations(MatrixValue result, org.apache.sysml.runtime.util.IndexRange range, boolean complementary)

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

SPARSITY_TURN_POINT

public static final double SPARSITY_TURN_POINT

See Also:

Constant Field Values

ULTRA_SPARSITY_TURN_POINT

public static final double ULTRA_SPARSITY_TURN_POINT

See Also:

Constant Field Values

ULTRA_SPARSITY_TURN_POINT2

public static final double ULTRA_SPARSITY_TURN_POINT2

See Also:

Constant Field Values

ULTRA_SPARSE_BLOCK_NNZ

public static final int ULTRA_SPARSE_BLOCK_NNZ

See Also:

Constant Field Values

DEFAULT_SPARSEBLOCK

public static final SparseBlock.Type DEFAULT_SPARSEBLOCK

DEFAULT_INPLACE_SPARSEBLOCK

public static final SparseBlock.Type DEFAULT_INPLACE_SPARSEBLOCK

MAX_SHALLOW_SERIALIZE_OVERHEAD

public static final double MAX_SHALLOW_SERIALIZE_OVERHEAD

See Also:

Constant Field Values

CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE

public static final boolean CONVERT_MCSR_TO_CSR_ON_DEEP_SERIALIZE

See Also:

Constant Field Values

HEADER_SIZE

public static final int HEADER_SIZE

See Also:

Constant Field Values

rlen

protected int rlen

clen

protected int clen

sparse

protected boolean sparse

nonZeros

protected long nonZeros

denseBlock

protected DenseBlock denseBlock

sparseBlock

protected SparseBlock sparseBlock

estimatedNNzsPerRow

protected int estimatedNNzsPerRow

Constructor Detail

MatrixBlock

public MatrixBlock()

MatrixBlock

public MatrixBlock(int rl,
                   int cl,
                   boolean sp)

MatrixBlock

public MatrixBlock(int rl,
                   int cl,
                   long estnnz)

MatrixBlock

public MatrixBlock(int rl,
                   int cl,
                   boolean sp,
                   long estnnz)

MatrixBlock

public MatrixBlock(MatrixBlock that)

MatrixBlock

public MatrixBlock(double val)

MatrixBlock

public MatrixBlock(int rl,
                   int cl,
                   double val)

MatrixBlock

public MatrixBlock(int rl,
                   int cl,
                   long nnz,
                   SparseBlock sblock)

Constructs a sparse MatrixBlock with a given instance of a SparseBlock

Parameters:

rl - number of rows

cl - number of columns

nnz - number of non zeroes

sblock - sparse block

MatrixBlock

public MatrixBlock(MatrixBlock that,
                   SparseBlock.Type stype,
                   boolean deep)

Method Detail

reset

public void reset()

Specified by:

reset in class MatrixValue

reset

public void reset(int rl,
                  int cl)

Specified by:

reset in class MatrixValue

reset

public void reset(int rl,
                  int cl,
                  long estnnz)

reset

public void reset(int rl,
                  int cl,
                  boolean sp)

Specified by:

reset in class MatrixValue

reset

public void reset(int rl,
                  int cl,
                  boolean sp,
                  long estnnz)

Specified by:

reset in class MatrixValue

reset

public void reset(int rl,
                  int cl,
                  double val)

Specified by:

reset in class MatrixValue

init

public void init(double[][] arr,
                 int r,
                 int c)

NOTE: This method is designed only for dense representation.

Parameters:

arr - 2d double array matrix

r - number of rows

c - number of columns

init

public void init(double[] arr,
                 int r,
                 int c)

NOTE: This method is designed only for dense representation.

Parameters:

arr - double array matrix

r - number of rows

c - number of columns

isAllocated

public boolean isAllocated()

allocateDenseBlock

public MatrixBlock allocateDenseBlock()

allocateBlock

public MatrixBlock allocateBlock()

allocateDenseBlock

public boolean allocateDenseBlock(boolean clearNNZ)

allocateSparseRowsBlock

public boolean allocateSparseRowsBlock()

allocateSparseRowsBlock

public boolean allocateSparseRowsBlock(boolean clearNNZ)

allocateAndResetSparseBlock

public void allocateAndResetSparseBlock(boolean clearNNZ,
                                        SparseBlock.Type stype)

allocateDenseBlockUnsafe

public void allocateDenseBlockUnsafe(int rl,
                                     int cl)

This should be called only in the read and write functions for CP This function should be called before calling any setValueDenseUnsafe()

Parameters:

rl - number of rows

cl - number of columns

cleanupBlock

public void cleanupBlock(boolean dense,
                         boolean sparse)

Allows to cleanup all previously allocated sparserows or denseblocks. This is for example required in reading a matrix with many empty blocks via distributed cache into in-memory list of blocks - not cleaning blocks from non-empty blocks would significantly increase the total memory consumption.

Parameters:

dense - if true, set dense block to null

sparse - if true, set sparse block to null

getNumRows

public int getNumRows()

Specified by:

getNumRows in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Specified by:

getNumRows in class MatrixValue

setNumRows

public void setNumRows(int r)

NOTE: setNumRows() and setNumColumns() are used only in ternaryInstruction (for contingency tables) and pmm for meta corrections.

Parameters:

r - number of rows

getNumColumns

public int getNumColumns()

Specified by:

getNumColumns in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Specified by:

getNumColumns in class MatrixValue

setNumColumns

public void setNumColumns(int c)

getNonZeros

public long getNonZeros()

Specified by:

getNonZeros in class MatrixValue

setNonZeros

public long setNonZeros(long nnz)

getSparsity

public double getSparsity()

getMatrixCharacteristics

public MatrixCharacteristics getMatrixCharacteristics()

isVector

public boolean isVector()

getLength

public long getLength()

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in class MatrixValue

isEmptyBlock

public boolean isEmptyBlock()

isEmptyBlock

public boolean isEmptyBlock(boolean safe)

getDenseBlock

public DenseBlock getDenseBlock()

getDenseBlockValues

public double[] getDenseBlockValues()

getSparseBlock

public SparseBlock getSparseBlock()

setSparseBlock

public void setSparseBlock(SparseBlock sblock)

getSparseBlockIterator

public Iterator<IJV> getSparseBlockIterator()

getSparseBlockIterator

public Iterator<IJV> getSparseBlockIterator(int rl,
                                            int ru)

getValue

public double getValue(int r,
                       int c)

Specified by:

getValue in class MatrixValue

setValue

public void setValue(int r,
                     int c,
                     double v)

Specified by:

setValue in class MatrixValue

quickGetValue

public double quickGetValue(int r,
                            int c)

quickSetValue

public void quickSetValue(int r,
                          int c,
                          double v)

getValueDenseUnsafe

public double getValueDenseUnsafe(int r,
                                  int c)

appendValue

public void appendValue(int r,
                        int c,
                        double v)

Append value is only used when values are appended at the end of each row for the sparse representation This can only be called, when the caller knows the access pattern of the block

Parameters:

r - row

c - column

v - value

appendRow

public void appendRow(int r,
                      SparseRow row)

appendRow

public void appendRow(int r,
                      SparseRow row,
                      boolean deep)

appendToSparse

public void appendToSparse(MatrixBlock that,
                           int rowoffset,
                           int coloffset)

appendToSparse

public void appendToSparse(MatrixBlock that,
                           int rowoffset,
                           int coloffset,
                           boolean deep)

appendRowToSparse

public void appendRowToSparse(SparseBlock dest,
                              MatrixBlock src,
                              int i,
                              int rowoffset,
                              int coloffset,
                              boolean deep)

sortSparseRows

public void sortSparseRows()

Sorts all existing sparse rows by column indexes.

sortSparseRows

public void sortSparseRows(int rl,
                           int ru)

Sorts all existing sparse rows in range [rl,ru) by column indexes.

Parameters:

rl - row lower bound, inclusive

ru - row upper bound, exclusive

minNonZero

public double minNonZero()

Utility function for computing the min non-zero value.

Returns:

minimum non-zero value

prod

public double prod()

Wrapper method for reduceall-product of a matrix.

Returns:

?

mean

public double mean()

Wrapper method for reduceall-mean of a matrix.

Returns:

?

min

public double min()

Wrapper method for reduceall-min of a matrix.

Returns:

?

max

public double max()

Wrapper method for reduceall-max of a matrix.

Returns:

?

sum

public double sum()

Wrapper method for reduceall-sum of a matrix.

Returns:

Sum of the values in the matrix.

sumSq

public double sumSq()

Wrapper method for reduceall-sumSq of a matrix.

Returns:

Sum of the squared values in the matrix.

isInSparseFormat

public boolean isInSparseFormat()

Returns the current representation (true for sparse).

Specified by:

isInSparseFormat in class MatrixValue

Returns:

true if sparse

isUltraSparse

public boolean isUltraSparse()

isUltraSparse

public boolean isUltraSparse(boolean checkNnz)

isUltraSparsePermutationMatrix

public boolean isUltraSparsePermutationMatrix()

evalSparseFormatInMemory

public boolean evalSparseFormatInMemory()

Evaluates if this matrix block should be in sparse format in memory. Note that this call does not change the representation - for this please call examSparsity.

Returns:

true if matrix block should be in sparse format in memory

evalSparseFormatOnDisk

public boolean evalSparseFormatOnDisk()

Evaluates if this matrix block should be in sparse format on disk. This applies to any serialized matrix representation, i.e., when writing to in-memory buffer pool pages or writing to local fs or hdfs.

Returns:

true if matrix block should be in sparse format on disk

examSparsity

public void examSparsity()

examSparsity

public void examSparsity(boolean allowCSR)

Evaluates if this matrix block should be in sparse format in memory. Depending on the current representation, the state of the matrix block is changed to the right representation if necessary. Note that this consumes for the time of execution memory for both representations.

Parameters:

allowCSR - allow CSR format on dense to sparse conversion

evalSparseFormatInMemory

public static boolean evalSparseFormatInMemory(long nrows,
                                               long ncols,
                                               long nnz)

Evaluates if a matrix block with the given characteristics should be in sparse format in memory.

Parameters:

nrows - number of rows

ncols - number of columns

nnz - number of non-zeros

Returns:

true if matrix block shold be in sparse format in memory

evalSparseFormatOnDisk

public static boolean evalSparseFormatOnDisk(long nrows,
                                             long ncols,
                                             long nnz)

Evaluates if a matrix block with the given characteristics should be in sparse format on disk (or in any other serialized representation).

Parameters:

nrows - number of rows

ncols - number of columns

nnz - number of non-zeros

Returns:

true if matrix block shold be in sparse format on disk

sparseToDense

public void sparseToDense()

recomputeNonZeros

public long recomputeNonZeros()

Recomputes and materializes the number of non-zero values of the entire matrix block.

Returns:

number of non-zeros

recomputeNonZeros

public long recomputeNonZeros(int rl,
                              int ru)

recomputeNonZeros

public long recomputeNonZeros(int rl,
                              int ru,
                              int cl,
                              int cu)

Recomputes the number of non-zero values of a specified range of the matrix block. NOTE: This call does not materialize the compute result in any form.

Parameters:

rl - row lower index, 0-based, inclusive

ru - row upper index, 0-based, inclusive

cl - column lower index, 0-based, inclusive

cu - column upper index, 0-based, inclusive

Returns:

the number of non-zero values

checkNonZeros

public void checkNonZeros()

Basic debugging primitive to check correctness of nnz. This method is not intended for production use.

checkSparseRows

public void checkSparseRows()

checkSparseRows

public void checkSparseRows(int rl,
                            int ru)

Basic debugging primitive to check sparse block column ordering. This method is not intended for production use.

Parameters:

rl - row lower bound (inclusive)

ru - row upper bound (exclusive)

copy

public void copy(MatrixValue thatValue)

Specified by:

copy in class MatrixValue

copy

public void copy(MatrixValue thatValue,
                 boolean sp)

Specified by:

copy in class MatrixValue

copyShallow

public MatrixBlock copyShallow(MatrixBlock that)

copy

public void copy(int rl,
                 int ru,
                 int cl,
                 int cu,
                 MatrixBlock src,
                 boolean awareDestNZ)

In-place copy of matrix src into the index range of the existing current matrix. Note that removal of existing nnz in the index range and nnz maintenance is only done if 'awareDestNZ=true',

Parameters:

rl - row lower index, 0-based

ru - row upper index, 0-based

cl - column lower index, 0-based

cu - column upper index, 0-based

src - matrix block

awareDestNZ - true, forces (1) to remove existing non-zeros in the index range of the destination if not present in src and (2) to internally maintain nnz false, assume empty index range in destination and do not maintain nnz (the invoker is responsible to recompute nnz after all copies are done)

merge

public void merge(org.apache.sysml.runtime.controlprogram.caching.CacheBlock that,
                  boolean appendOnly)

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Merge the given block into the current block. Both blocks needs to be of equal dimensions and contain disjoint non-zero cells.

Specified by:

merge in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Parameters:

that - cache block

appendOnly - ?

merge

public void merge(MatrixBlock that,
                  boolean appendOnly)

Merge disjoint: merges all non-zero values of the given input into the current matrix block. Note that this method does NOT check for overlapping entries; it's the callers reponsibility of ensuring disjoint matrix blocks. The appendOnly parameter is only relevant for sparse target blocks; if true, we only append values and do not sort sparse rows for each call; this is useful whenever we merge iterators of matrix blocks into one target block.

Parameters:

that - matrix block

appendOnly - ?

merge

public void merge(MatrixBlock that,
                  boolean appendOnly,
                  boolean par)

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Throws:

IOException

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Throws:

IOException

readExternal

public void readExternal(ObjectInput is)
                  throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.

Specified by:

readExternal in interface Externalizable

Parameters:

is - object input

Throws:

IOException - if IOException occurs

writeExternal

public void writeExternal(ObjectOutput os)
                   throws IOException

Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.

Specified by:

writeExternal in interface Externalizable

Parameters:

os - object output

Throws:

IOException - if IOException occurs

getExactSizeOnDisk

public long getExactSizeOnDisk()

NOTE: The used estimates must be kept consistent with the respective write functions.

Returns:

exact size on disk

estimateSizeInMemory

public long estimateSizeInMemory()

estimateSizeInMemory

public static long estimateSizeInMemory(long nrows,
                                        long ncols,
                                        double sparsity)

estimateSizeDenseInMemory

public static long estimateSizeDenseInMemory(long nrows,
                                             long ncols)

estimateSizeSparseInMemory

public static long estimateSizeSparseInMemory(long nrows,
                                              long ncols,
                                              double sparsity)

estimateSizeSparseInMemory

public static long estimateSizeSparseInMemory(long nrows,
                                              long ncols,
                                              double sparsity,
                                              SparseBlock.Type stype)

estimateSizeOnDisk

public long estimateSizeOnDisk()

estimateSizeOnDisk

public static long estimateSizeOnDisk(long nrows,
                                      long ncols,
                                      long nnz)

estimateSparsityOnAggBinary

public static MatrixBlock.SparsityEstimate estimateSparsityOnAggBinary(MatrixBlock m1,
                                                                       MatrixBlock m2,
                                                                       org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)

getInMemorySize

public long getInMemorySize()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Get the in-memory size in bytes of the cache block.

Specified by:

getInMemorySize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Returns:

in-memory size in bytes of cache block

getExactSerializedSize

public long getExactSerializedSize()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Get the exact serialized size in bytes of the cache block.

Specified by:

getExactSerializedSize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Returns:

exact serialized size in bytes of cache block

isShallowSerialize

public boolean isShallowSerialize()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Returns:

true if shallow serialized

isShallowSerialize

public boolean isShallowSerialize(boolean inclConvert)

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Indicates if the cache block is subject to shallow serialized, which is generally true if in-memory size and serialized size are almost identical allowing to avoid unnecessary deep serialize.

Specified by:

isShallowSerialize in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Parameters:

inclConvert - if true report blocks as shallow serialize that are currently not amenable but can be brought into an amenable form via toShallowSerializeBlock.

Returns:

true if shallow serialized

toShallowSerializeBlock

public void toShallowSerializeBlock()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Converts a cache block that is not shallow serializable into a form that is shallow serializable. This methods has no affect if the given cache block is not amenable.

Specified by:

toShallowSerializeBlock in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

compactEmptyBlock

public void compactEmptyBlock()

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Free unnecessarily allocated empty block.

Specified by:

compactEmptyBlock in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

scalarOperations

public MatrixValue scalarOperations(org.apache.sysml.runtime.matrix.operators.ScalarOperator op,
                                    MatrixValue result)

Specified by:

scalarOperations in class MatrixValue

unaryOperations

public MatrixValue unaryOperations(org.apache.sysml.runtime.matrix.operators.UnaryOperator op,
                                   MatrixValue result)

Specified by:

unaryOperations in class MatrixValue

binaryOperations

public MatrixValue binaryOperations(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue,
                                    MatrixValue result)

Specified by:

binaryOperations in class MatrixValue

binaryOperationsInPlace

public void binaryOperationsInPlace(org.apache.sysml.runtime.matrix.operators.BinaryOperator op,
                                    MatrixValue thatValue)

Specified by:

binaryOperationsInPlace in class MatrixValue

ternaryOperations

public MatrixBlock ternaryOperations(org.apache.sysml.runtime.matrix.operators.TernaryOperator op,
                                     MatrixBlock m2,
                                     MatrixBlock m3,
                                     MatrixBlock ret)

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue correction,
                                 MatrixValue newWithCorrection,
                                 boolean deep)

Specified by:

incrementalAggregate in class MatrixValue

incrementalAggregate

public void incrementalAggregate(org.apache.sysml.runtime.matrix.operators.AggregateOperator aggOp,
                                 MatrixValue newWithCorrection)

Specified by:

incrementalAggregate in class MatrixValue

reorgOperations

public MatrixValue reorgOperations(org.apache.sysml.runtime.matrix.operators.ReorgOperator op,
                                   MatrixValue ret,
                                   int startRow,
                                   int startColumn,
                                   int length)

Specified by:

reorgOperations in class MatrixValue

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret)

append

public MatrixBlock append(MatrixBlock that,
                          MatrixBlock ret,
                          boolean cbind)

append

public MatrixBlock append(MatrixBlock[] that,
                          MatrixBlock ret,
                          boolean cbind)

naryOperations

public static MatrixBlock naryOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                                         MatrixBlock[] matrices,
                                         org.apache.sysml.runtime.instructions.cp.ScalarObject[] scalars,
                                         MatrixBlock ret)

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype)

transposeSelfMatrixMultOperations

public MatrixBlock transposeSelfMatrixMultOperations(MatrixBlock out,
                                                     org.apache.sysml.lops.MMTSJ.MMTSJType tstype,
                                                     int k)

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype)

chainMatrixMultOperations

public MatrixBlock chainMatrixMultOperations(MatrixBlock v,
                                             MatrixBlock w,
                                             MatrixBlock out,
                                             org.apache.sysml.lops.MapMultChain.ChainType ctype,
                                             int k)

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val)

permutationMatrixMultOperations

public void permutationMatrixMultOperations(MatrixValue m2Val,
                                            MatrixValue out1Val,
                                            MatrixValue out2Val,
                                            int k)

leftIndexingOperations

public final MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix,
                                                org.apache.sysml.runtime.util.IndexRange ixrange,
                                                MatrixBlock ret,
                                                org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)

leftIndexingOperations

public MatrixBlock leftIndexingOperations(MatrixBlock rhsMatrix,
                                          int rl,
                                          int ru,
                                          int cl,
                                          int cu,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)

leftIndexingOperations

public MatrixBlock leftIndexingOperations(org.apache.sysml.runtime.instructions.cp.ScalarObject scalar,
                                          int rl,
                                          int cl,
                                          MatrixBlock ret,
                                          org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType update)

Explicitly allow left indexing for scalars. Note: This operation is now 0-based. * Operations to be performed: 1) result=this; 2) result[row,column] = scalar.getDoubleValue();

Parameters:

scalar - scalar object

rl - row lower

cl - column lower

ret - ?

update - ?

Returns:

matrix block

slice

public MatrixBlock slice(org.apache.sysml.runtime.util.IndexRange ixrange,
                         MatrixBlock ret)

slice

public MatrixBlock slice(int rl,
                         int ru)

slice

public MatrixBlock slice(int rl,
                         int ru,
                         int cl,
                         int cu,
                         org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret)

Description copied from interface: org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Slice a sub block out of the current block and write into the given output block. This method returns the passed instance if not null.

Specified by:

slice in interface org.apache.sysml.runtime.controlprogram.caching.CacheBlock

Parameters:

rl - row lower

ru - row upper

cl - column lower

cu - column upper

ret - cache block

Returns:

sub-block of cache block

slice

public MatrixBlock slice(int rl,
                         int ru,
                         int cl,
                         int cu,
                         boolean deep,
                         org.apache.sysml.runtime.controlprogram.caching.CacheBlock ret)

Method to perform rightIndex operation for a given lower and upper bounds in row and column dimensions. Extracted submatrix is returned as "result". Note: This operation is now 0-based.

Parameters:

rl - row lower

ru - row upper

cl - column lower

cu - column upper

deep - should perform deep copy

ret - output matrix block

Returns:

matrix block output matrix block

slice

public void slice(ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                  org.apache.sysml.runtime.util.IndexRange range,
                  int rowCut,
                  int colCut,
                  int normalBlockRowFactor,
                  int normalBlockColFactor,
                  int boundaryRlen,
                  int boundaryClen)

Specified by:

slice in class MatrixValue

append

public void append(MatrixValue v2,
                   ArrayList<org.apache.sysml.runtime.matrix.mapred.IndexedMatrixValue> outlist,
                   int blockRowFactor,
                   int blockColFactor,
                   boolean cbind,
                   boolean m2IsLast,
                   int nextNCol)

Specified by:

append in class MatrixValue

zeroOutOperations

public MatrixValue zeroOutOperations(MatrixValue result,
                                     org.apache.sysml.runtime.util.IndexRange range,
                                     boolean complementary)

Specified by:

zeroOutOperations in class MatrixValue

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn)

Specified by:

aggregateUnaryOperations in class MatrixValue

aggregateUnaryOperations

public MatrixValue aggregateUnaryOperations(org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator op,
                                            MatrixValue result,
                                            int blockingFactorRow,
                                            int blockingFactorCol,
                                            MatrixIndexes indexesIn,
                                            boolean inCP)

Specified by:

aggregateUnaryOperations in class MatrixValue

dropLastRowsOrColumns

public void dropLastRowsOrColumns(org.apache.sysml.lops.PartialAggregate.CorrectionLocationType correctionLocation)

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op)

cmOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object cmOperations(org.apache.sysml.runtime.matrix.operators.CMOperator op,
                                                                           MatrixBlock weights)

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that)

covOperations

public org.apache.sysml.runtime.instructions.cp.CM_COV_Object covOperations(org.apache.sysml.runtime.matrix.operators.COVOperator op,
                                                                            MatrixBlock that,
                                                                            MatrixBlock weights)

sortOperations

public MatrixValue sortOperations(MatrixValue weights,
                                  MatrixValue result)

interQuartileMean

public double interQuartileMean()

computeIQMCorrection

public static double computeIQMCorrection(double sum,
                                          double sum_wt,
                                          double q25Part,
                                          double q25Val,
                                          double q75Part,
                                          double q75Val)

pickValues

public MatrixValue pickValues(MatrixValue quantiles,
                              MatrixValue ret)

median

public double median()

pickValue

public double pickValue(double quantile)

pickValue

public double pickValue(double quantile,
                        boolean average)

sumWeightForQuantile

public double sumWeightForQuantile()

In a given two column matrix, the second column denotes weights. This function computes the total weight

Returns:

sum weight for quantile

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixIndexes m1Index,
                                             MatrixBlock m1,
                                             MatrixIndexes m2Index,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)

aggregateBinaryOperations

public MatrixBlock aggregateBinaryOperations(MatrixBlock m1,
                                             MatrixBlock m2,
                                             MatrixBlock ret,
                                             org.apache.sysml.runtime.matrix.operators.AggregateBinaryOperator op)

aggregateTernaryOperations

public MatrixBlock aggregateTernaryOperations(MatrixBlock m1,
                                              MatrixBlock m2,
                                              MatrixBlock m3,
                                              MatrixBlock ret,
                                              org.apache.sysml.runtime.matrix.operators.AggregateTernaryOperator op,
                                              boolean inCP)

uaggouterchainOperations

public MatrixBlock uaggouterchainOperations(MatrixBlock mbLeft,
                                            MatrixBlock mbRight,
                                            MatrixBlock mbOut,
                                            org.apache.sysml.runtime.matrix.operators.BinaryOperator bOp,
                                            org.apache.sysml.runtime.matrix.operators.AggregateUnaryOperator uaggOp)

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op)

Invocation from CP instructions. The aggregate is computed on the groups object against target and weights. Notes: * The computed number of groups is reused for multiple invocations with different target. * This implementation supports that the target is passed as column or row vector, in case of row vectors we also use sparse-safe implementations for sparse safe aggregation operators.

Parameters:

tgt - ?

wghts - ?

ret - ?

ngroups - ?

op - operator

Returns:

matrix block

groupedAggOperations

public MatrixBlock groupedAggOperations(MatrixValue tgt,
                                        MatrixValue wghts,
                                        MatrixValue ret,
                                        int ngroups,
                                        org.apache.sysml.runtime.matrix.operators.Operator op,
                                        int k)

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn,
                                         MatrixBlock select)

removeEmptyOperations

public MatrixBlock removeEmptyOperations(MatrixBlock ret,
                                         boolean rows,
                                         boolean emptyReturn)

rexpandOperations

public MatrixBlock rexpandOperations(MatrixBlock ret,
                                     double max,
                                     boolean rows,
                                     boolean cast,
                                     boolean ignore,
                                     int k)

replaceOperations

public MatrixValue replaceOperations(MatrixValue result,
                                     double pattern,
                                     double replacement)

Specified by:

replaceOperations in class MatrixValue

extractTriangular

public MatrixBlock extractTriangular(MatrixBlock ret,
                                     boolean lower,
                                     boolean diag,
                                     boolean values)

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             double scalarThat,
                             MatrixValue that2Val,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

D = ctable(A,v2,W) this <- A; scalarThat <- v2; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (i3,j3,w) from input3 (that2)

Specified by:

ctableOperations in class MatrixValue

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             double scalarThat,
                             double scalarThat2,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

D = ctable(A,v2,w) this <- A; scalar_that <- v2; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (v2) from sclar_input2 (scalarThat) (w) from scalar_input3 (scalarThat2)

Specified by:

ctableOperations in class MatrixValue

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixIndexes ix1,
                             double scalarThat,
                             boolean left,
                             int brlen,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Specific ctable case of ctable(seq(...),X), where X is the only matrix input. The 'left' input parameter specifies if the seq appeared on the left, otherwise it appeared on the right.

Specified by:

ctableOperations in class MatrixValue

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue thatVal,
                             double scalarThat2,
                             boolean ignoreZeros,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

D = ctable(A,B,w) this <- A; that <- B; scalar_that2 <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2) NOTE: This method supports both vectors and matrices. In case of matrices and ignoreZeros=true we can also use a sparse-safe implementation

Specified by:

ctableOperations in class MatrixValue

ctableSeqOperations

public MatrixBlock ctableSeqOperations(MatrixValue thatMatrix,
                                       double thatScalar,
                                       MatrixBlock resultBlock)

D = ctable(seq,A,w) this <- seq; thatMatrix <- A; thatScalar <- w; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (w) from scalar_input3 (scalarThat2)

Parameters:

thatMatrix - matrix value

thatScalar - scalar double

resultBlock - result matrix block

Returns:

resultBlock

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue thatVal,
                             MatrixValue that2Val,
                             CTableMap resultMap)

D = ctable(A,B,W) this <- A; that <- B; that2 <- W; result <- D (i1,j1,v1) from input1 (this) (i1,j1,v2) from input2 (that) (i1,j1,w) from input3 (that2)

Parameters:

op - operator

thatVal - matrix value 1

that2Val - matrix value 2

resultMap - table map

ctableOperations

public void ctableOperations(org.apache.sysml.runtime.matrix.operators.Operator op,
                             MatrixValue thatVal,
                             MatrixValue that2Val,
                             CTableMap resultMap,
                             MatrixBlock resultBlock)

Specified by:

ctableOperations in class MatrixValue

quaternaryOperations

public MatrixBlock quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixBlock um,
                                        MatrixBlock vm,
                                        MatrixBlock wm,
                                        MatrixBlock out)

quaternaryOperations

public MatrixBlock quaternaryOperations(org.apache.sysml.runtime.matrix.operators.QuaternaryOperator qop,
                                        MatrixBlock U,
                                        MatrixBlock V,
                                        MatrixBlock wm,
                                        MatrixBlock out,
                                        int k)

randOperations

public static MatrixBlock randOperations(int rows,
                                         int cols,
                                         double sparsity,
                                         double min,
                                         double max,
                                         String pdf,
                                         long seed)

Function to generate the random matrix with specified dimensions (block sizes are not specified).

Parameters:

rows - number of rows

cols - number of columns

sparsity - sparsity as a percentage

min - minimum value

max - maximum value

pdf - pdf

seed - random seed

Returns:

matrix block

randOperations

public static MatrixBlock randOperations(int rows,
                                         int cols,
                                         double sparsity,
                                         double min,
                                         double max,
                                         String pdf,
                                         long seed,
                                         int k)

Function to generate the random matrix with specified dimensions (block sizes are not specified).

Parameters:

rows - number of rows

cols - number of columns

sparsity - sparsity as a percentage

min - minimum value

max - maximum value

pdf - pdf

seed - random seed

k - ?

Returns:

matrix block

randOperations

public static MatrixBlock randOperations(RandomMatrixGenerator rgen,
                                         long seed)

Function to generate the random matrix with specified dimensions and block dimensions.

Parameters:

rgen - random matrix generator

seed - seed value

Returns:

matrix block

randOperations

public static MatrixBlock randOperations(RandomMatrixGenerator rgen,
                                         long seed,
                                         int k)

Function to generate the random matrix with specified dimensions and block dimensions.

Parameters:

rgen - random matrix generator

seed - seed value

k - ?

Returns:

matrix block

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed)

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

Returns:

matrix block

randOperationsInPlace

public MatrixBlock randOperationsInPlace(RandomMatrixGenerator rgen,
                                         org.apache.commons.math3.random.Well1024a bigrand,
                                         long bSeed,
                                         int k)

Function to generate a matrix of random numbers. This is invoked both from CP as well as from MR. In case of CP, it generates an entire matrix block-by-block. A bigrand is passed so that block-level seeds are generated internally. In case of MR, it generates a single block for given block-level seed bSeed. When pdf="uniform", cell values are drawn from uniform distribution in range [min,max]. When pdf="normal", cell values are drawn from standard normal distribution N(0,1). The range of generated values will always be (-Inf,+Inf).

Parameters:

rgen - random matrix generator

bigrand - ?

bSeed - seed value

k - ?

Returns:

matrix block

seqOperations

public static MatrixBlock seqOperations(double from,
                                        double to,
                                        double incr)

Method to generate a sequence according to the given parameters. The generated sequence is always in dense format. Both end points specified from and to must be included in the generated sequence i.e., [from,to] both inclusive. Note that, to is included only if (to-from) is perfectly divisible by incr. For example, seq(0,1,0.5) generates (0.0 0.5 1.0) whereas seq(0,1,0.6) generates (0.0 0.6) but not (0.0 0.6 1.0)

Parameters:

from - ?

to - ?

incr - ?

Returns:

matrix block

seqOperationsInPlace

public MatrixBlock seqOperationsInPlace(double from,
                                        double to,
                                        double incr)

sampleOperations

public static MatrixBlock sampleOperations(long range,
                                           int size,
                                           boolean replace,
                                           long seed)

isThreadSafe

public boolean isThreadSafe()

Indicates if concurrent modifications of disjoint rows are thread-safe.

Returns:

true if thread-safe

isThreadSafe

public static boolean isThreadSafe(boolean sparse)

Indicates if concurrent modifications of disjoint rows are thread-safe.

Parameters:

sparse - true if sparse

Returns:

true if ?

checkNaN

public void checkNaN()

Checks for existing NaN values in the matrix block.

Throws:

DMLRuntimeException - if the blocks contains at least one NaN.

compareTo

public int compareTo(Object arg0)

Specified by:

compareTo in interface Comparable

equals

public boolean equals(Object arg0)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object