org.apache.sysds.runtime.compress.colgroup

Class ColGroup

java.lang.Object

org.apache.sysds.runtime.compress.colgroup.ColGroup

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

ColGroupUncompressed, ColGroupValue

public abstract class ColGroup
extends Object
implements Serializable

Class that stores information about a column group within a compressed matrix block. There are subclasses specific to each compression type.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ColGroup.CompressionType Public Group types supported Note For instance DDC is called DDC not DDC1, or DDC2 which is a specific subtype of the DDC.

Method Summary

Modifier and Type	Method and Description
abstract void	countNonZerosPerRow(int[] rnnz, int rl, int ru) Count the number of non-zeros per row
abstract void	decompressToBlock(MatrixBlock target, int colpos) Decompress to block.
abstract void	decompressToBlock(MatrixBlock target, int[] colIndexTargets) Decompress the contents of this column group into uncompressed packed columns
abstract void	decompressToBlock(MatrixBlock target, int rl, int ru) Decompress the contents of this column group into the specified full matrix block.
abstract long	estimateInMemorySize() Note: Must be overridden by child classes to account for additional data and metadata
abstract double	get(int r, int c) Get the value at a global row/column position.
int	getColIndex(int colNum) Obtain a column index value.
int[]	getColIndices() Obtain the offsets of the columns in the matrix block that make up the group
abstract ColGroup.CompressionType	getCompType() Obtain the compression type.
abstract long	getExactSizeOnDisk() Returns the exact serialized size of column group.
abstract boolean	getIfCountsType() Returns true if in the getValuesAsBlock method returns values in groups (that needs to be counted) or individually potentially repeated values
abstract Iterator<IJV>	getIterator(int rl, int ru, boolean inclZeros, boolean rowMajor) Create a column group iterator for a row index range.
int	getNumCols() Obtain the number of columns in this column group.
int	getNumRows() Get number of rows contained in the ColGroup.
abstract org.apache.sysds.runtime.compress.colgroup.ColGroup.ColGroupRowIterator	getRowIterator(int rl, int ru) Create a dense row iterator for a row index range.
abstract double[]	getValues() Get all the values in the colGroup.
abstract MatrixBlock	getValuesAsBlock() Returns the ColGroup as a MatrixBlock.
abstract boolean	isLossy() Is Lossy
abstract void	leftMultByMatrix(double[] matrix, double[] result, int numVals, double[] values, int numRows, int numCols, int rl, int ru, int vOff) Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e.
abstract void	leftMultByRowVector(double[] vector, double[] result, int numVals) Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e.
abstract void	leftMultByRowVector(double[] vector, double[] result, int numVals, double[] values) Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e.
abstract void	readFields(DataInput in) Deserialize column group from data input.
abstract void	rightMultByMatrix(double[] matrix, double[] result, int numVals, double[] values, int rl, int ru, int vOff)
abstract void	rightMultByVector(double[] vector, double[] c, int rl, int ru, double[] dictVals) Multiply the slice of the matrix that this column group represents by a vector on the right.
abstract ColGroup	scalarOperation(ScalarOperator op) Perform the specified scalar operation directly on the compressed column group, without decompressing individual cells if possible.
void	shiftColIndices(int offset)
abstract void	unaryAggregateOperations(AggregateUnaryOperator op, double[] c) Unary Aggregate operator, since aggregate operators require new object output, the output becomes an uncompressed matrix.
abstract void	unaryAggregateOperations(AggregateUnaryOperator op, double[] c, int rl, int ru) Unary Aggregate operator, since aggregate operators require new object output, the output becomes an uncompressed matrix.
abstract void	write(DataOutput out) Serializes column group to data output.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getColIndices

public int[] getColIndices()

Obtain the offsets of the columns in the matrix block that make up the group

Returns:

offsets of the columns in the matrix block that make up the group

getColIndex

public int getColIndex(int colNum)

Obtain a column index value.

Parameters:

colNum - column number

Returns:

column index value

getNumRows

public int getNumRows()

Get number of rows contained in the ColGroup.

Returns:

An integer that is the number of rows.

getNumCols

public int getNumCols()

Obtain the number of columns in this column group.

Returns:

number of columns in this column group

getCompType

public abstract ColGroup.CompressionType getCompType()

Obtain the compression type.

Returns:

How the elements of the column group are compressed.

shiftColIndices

public void shiftColIndices(int offset)

estimateInMemorySize

public abstract long estimateInMemorySize()

Note: Must be overridden by child classes to account for additional data and metadata

Returns:

an upper bound on the number of bytes used to store this ColGroup in memory.

decompressToBlock

public abstract void decompressToBlock(MatrixBlock target,
                                       int rl,
                                       int ru)

Decompress the contents of this column group into the specified full matrix block.

Parameters:

target - a matrix block where the columns covered by this column group have not yet been filled in.

rl - row lower

ru - row upper

decompressToBlock

public abstract void decompressToBlock(MatrixBlock target,
                                       int[] colIndexTargets)

Decompress the contents of this column group into uncompressed packed columns

Parameters:

target - a dense matrix block. The block must have enough space to hold the contents of this column group.

colIndexTargets - array that maps column indices in the original matrix block to columns of target.

decompressToBlock

public abstract void decompressToBlock(MatrixBlock target,
                                       int colpos)

Decompress to block.

Parameters:

target - dense output vector

colpos - column to decompress, error if larger or equal numCols

write

public abstract void write(DataOutput out)
                    throws IOException

Serializes column group to data output.

Parameters:

out - data output

Throws:

IOException - if IOException occurs

readFields

public abstract void readFields(DataInput in)
                         throws IOException

Deserialize column group from data input.

Parameters:

in - data input

Throws:

IOException - if IOException occurs

getExactSizeOnDisk

public abstract long getExactSizeOnDisk()

Returns the exact serialized size of column group. This can be used for example for buffer preallocation.

Returns:

exact serialized size for column group

get

public abstract double get(int r,
                           int c)

Get the value at a global row/column position.

Parameters:

r - row

c - column

Returns:

value at the row/column position

getValues

public abstract double[] getValues()

Get all the values in the colGroup. Note that this is only the stored values not the way they are stored. Making the output a list of values used in that colGroup not the actual full column.

Returns:

a double list of values.

getValuesAsBlock

public abstract MatrixBlock getValuesAsBlock()

Returns the ColGroup as a MatrixBlock. Used as a fall back solution in case a operation is not supported. Use in connection to getIfCountsType to get if the values are repeated.

Returns:

Matrix Block of the contained Values. Possibly contained in groups.

getIfCountsType

public abstract boolean getIfCountsType()

Returns true if in the getValuesAsBlock method returns values in groups (that needs to be counted) or individually potentially repeated values

Returns:

boolean

rightMultByVector

public abstract void rightMultByVector(double[] vector,
                                       double[] c,
                                       int rl,
                                       int ru,
                                       double[] dictVals)

Multiply the slice of the matrix that this column group represents by a vector on the right.

Parameters:

vector - vector to multiply by (tall vector)

c - accumulator for holding the result

rl - row lower

ru - row upper if the internal SystemML code that performs the multiplication experiences an error

dictVals - The dictionary values materialized

rightMultByMatrix

public abstract void rightMultByMatrix(double[] matrix,
                                       double[] result,
                                       int numVals,
                                       double[] values,
                                       int rl,
                                       int ru,
                                       int vOff)

leftMultByRowVector

public abstract void leftMultByRowVector(double[] vector,
                                         double[] result,
                                         int numVals)

Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e. its size now is 1xn).

Parameters:

vector - row vector

result - matrix block result

numVals - The Number of values contained in the Column.

leftMultByRowVector

public abstract void leftMultByRowVector(double[] vector,
                                         double[] result,
                                         int numVals,
                                         double[] values)

Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e. its size now is 1xn).

Parameters:

vector - row vector

result - matrix block result

numVals - The Number of values contained in the Column.

values - The materialized list of values contained in the dictionary.

leftMultByMatrix

public abstract void leftMultByMatrix(double[] matrix,
                                      double[] result,
                                      int numVals,
                                      double[] values,
                                      int numRows,
                                      int numCols,
                                      int rl,
                                      int ru,
                                      int vOff)

Multiply the slice of the matrix that this column group represents by a row vector on the left (the original column vector is assumed to be transposed already i.e. its size now is 1xn).

Parameters:

matrix - matrix to left multiply

result - matrix block result

numVals - The Number of values contained in the Column.

values - The materialized list of values contained in the dictionary.

numRows - The number of rows in the matrix input

numCols - The number of columns in the colGroups parent matrix.

rl - The row to start the matrix multiplication from

ru - The row to stop the matrix multiplication at.

vOff - The offset into the first argument matrix to start at.

scalarOperation

public abstract ColGroup scalarOperation(ScalarOperator op)

Perform the specified scalar operation directly on the compressed column group, without decompressing individual cells if possible.

Parameters:

op - operation to perform

Returns:

version of this column group with the operation applied

unaryAggregateOperations

public abstract void unaryAggregateOperations(AggregateUnaryOperator op,
                                              double[] c)

Unary Aggregate operator, since aggregate operators require new object output, the output becomes an uncompressed matrix.

Parameters:

op - The operator used

c - Rhe output matrix block.

unaryAggregateOperations

public abstract void unaryAggregateOperations(AggregateUnaryOperator op,
                                              double[] c,
                                              int rl,
                                              int ru)

Unary Aggregate operator, since aggregate operators require new object output, the output becomes an uncompressed matrix.

Parameters:

op - The operator used

c - The output matrix block.

rl - The Starting Row to do aggregation from

ru - The last Row to do aggregation to (not included)

getIterator

public abstract Iterator<IJV> getIterator(int rl,
                                          int ru,
                                          boolean inclZeros,
                                          boolean rowMajor)

Create a column group iterator for a row index range.

Parameters:

rl - row lower index, inclusive

ru - row upper index, exclusive

inclZeros - include zero values into scope of iterator

rowMajor - use a row major iteration order

Returns:

an iterator instance

getRowIterator

public abstract org.apache.sysds.runtime.compress.colgroup.ColGroup.ColGroupRowIterator getRowIterator(int rl,
                                                                                                       int ru)

Create a dense row iterator for a row index range. This iterator implies the inclusion of zeros and row-major iteration order.

Parameters:

rl - row lower index, inclusive

ru - row upper index, exclusive

Returns:

an iterator instance

countNonZerosPerRow

public abstract void countNonZerosPerRow(int[] rnnz,
                                         int rl,
                                         int ru)

Count the number of non-zeros per row

Parameters:

rnnz - non-zeros per row

rl - row lower bound, inclusive

ru - row upper bound, exclusive

isLossy

public abstract boolean isLossy()

Is Lossy

Returns:

returns if the ColGroup is compressed in a lossy manner.