org.apache.sysds.runtime.compress.colgroup.dictionary

Class ADictionary

java.lang.Object

org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

Dictionary, MatrixBlockDictionary, QDictionary

public abstract class ADictionary
extends Object
implements Serializable

This dictionary class aims to encapsulate the storage and operations over unique tuple values of a column group.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ADictionary()

Method Summary

Modifier and Type	Method and Description
abstract void	addMaxAndMin(double[] ret, int[] colIndexes) This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable.
abstract void	addToEntry(Dictionary d, int fr, int to, int nCol) Copies and adds the dictionary entry from this dictionary to the d dictionary
abstract double	aggregate(double init, Builtin fn) Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.
abstract void	aggregateCols(double[] c, Builtin fn, int[] colIndexes) Aggregates the columns into the target double array provided.
abstract double[]	aggregateTuples(Builtin fn, int nCol) Aggregate all entries in the rows.
abstract ADictionary	apply(ScalarOperator op) Applies the scalar operation on the dictionary.
ADictionary	applyBinaryRowOp(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes, boolean left) Apply binary row operation on this dictionary.
abstract ADictionary	applyBinaryRowOpLeft(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes) Apply binary row operation on this dictionary on the left side.
abstract ADictionary	applyBinaryRowOpRight(BinaryOperator op, double[] v, boolean sparseSafe, int[] colIndexes) Apply binary row operation on this dictionary on the right side.
abstract ADictionary	applyScalarOp(ScalarOperator op, double newVal, int numCols) Applies the scalar operation on the dictionary.
abstract ADictionary	clone() Returns a deep clone of the dictionary.
abstract ADictionary	cloneAndExtend(int len) Clone the dictionary, and extend size of the dictionary by a given length
abstract void	colProduct(double[] res, int[] counts, int[] colIndexes)
abstract void	colSum(double[] c, int[] counts, int[] colIndexes, boolean square) Get the column sum of the values contained in the dictionary
abstract double[]	colSum(int[] counts, int nCol) get the column sum of this dictionary only.
abstract boolean	containsValue(double pattern) Detect if the dictionary contains a specific value.
abstract MatrixBlockDictionary	getAsMatrixBlockDictionary(int nCol) Get this dictionary as a matrixBlock dictionary.
abstract long	getExactSizeOnDisk() Calculate the space consumption if the dictionary is stored on disk.
abstract long	getInMemorySize() Returns the memory usage of the dictionary.
double[]	getMostCommonTuple(int[] counts, int nCol) Get the most common tuple element contained in the dictionary returns null if that tuple is all zero values.
abstract long	getNumberNonZeros(int[] counts, int nCol) Calculate the number of non zeros in the dictionary.
abstract int	getNumberOfValues(int ncol) Get the number of distinct tuples given that the column group has n columns
abstract String	getString(int colIndexes) Get a string representation of the dictionary, that considers the layout of the data.
abstract double[]	getTuple(int index, int nCol) Get the values contained in a specific tuple of the dictionary.
abstract double	getValue(int i) Get Specific value contained in the dictionary at index.
abstract double[]	getValues() Get all the values contained in the dictionary as a linearized double array.
abstract boolean	isLossy() Specify if the Dictionary is lossy.
abstract ADictionary	preaggValuesFromDense(int numVals, int[] colIndexes, int[] aggregateColumns, double[] b, int cut) Pre Aggregate values for right Matrix Multiplication.
abstract double	product(int[] counts, int nCol)
abstract ADictionary	reExpandColumns(int max) return a new Dictionary that have re expanded all values, based on the entries already contained.
abstract ADictionary	replace(double pattern, double replace, int nCol) Make a copy of the values, and replace all values that match pattern with replacement value.
abstract ADictionary	replaceZeroAndExtend(double replace, int nCol)
abstract ADictionary	scaleTuples(int[] scaling, int nCol) Scale all tuples contained in the dictionary by the scaling factor given in the int list.
abstract ADictionary	sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns) Modify the dictionary by removing columns not within the index range.
abstract ADictionary	subtractTuple(double[] tuple) Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.
abstract double	sum(int[] counts, int nCol) Get the sum of the values contained in the dictionary
abstract double[]	sumAllRowsToDouble(boolean square, int nrColumns) Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
abstract double	sumRow(int k, boolean square, int nrColumns) Sum the values at a specific row.
abstract double	sumsq(int[] counts, int nCol) Get the square sum of the values contained in the dictionary
abstract void	write(DataOutput out) Write the dictionary to a DataOutput.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ADictionary

public ADictionary()

Method Detail

getValues

public abstract double[] getValues()

Get all the values contained in the dictionary as a linearized double array.

Returns:

linearized double array

getValue

public abstract double getValue(int i)

Get Specific value contained in the dictionary at index.

Parameters:

i - The index to extract the value from

Returns:

The value contained at the index

getInMemorySize

public abstract long getInMemorySize()

Returns the memory usage of the dictionary.

Returns:

a long value in number of bytes for the dictionary.

aggregate

public abstract double aggregate(double init,
                                 Builtin fn)

Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.

Parameters:

init - The initial Value, in cases such as Max value, this could be -infinity

fn - The Function to apply to values

Returns:

The aggregated value as a double.

aggregateTuples

public abstract double[] aggregateTuples(Builtin fn,
                                         int nCol)

Aggregate all entries in the rows.

Parameters:

fn - The aggregate function

nCol - The number of columns contained in the dictionary.

Returns:

Aggregates for this dictionary tuples.

apply

public abstract ADictionary apply(ScalarOperator op)

Applies the scalar operation on the dictionary. Note that this operation modifies the underlying data, and normally require a copy of the original Dictionary to preserve old objects.

Parameters:

op - The operator to apply to the dictionary values.

Returns:

this dictionary with modified values.

applyScalarOp

public abstract ADictionary applyScalarOp(ScalarOperator op,
                                          double newVal,
                                          int numCols)

Applies the scalar operation on the dictionary. The returned dictionary should contain a new instance of the underlying data. Therefore it will not modify the previous object.

Parameters:

op - The operator to apply to the dictionary values.

newVal - The value to append to the dictionary.

numCols - The number of columns stored in the dictionary.

Returns:

Another dictionary with modified values.

applyBinaryRowOp

public ADictionary applyBinaryRowOp(BinaryOperator op,
                                    double[] v,
                                    boolean sparseSafe,
                                    int[] colIndexes,
                                    boolean left)

Apply binary row operation on this dictionary.

Parameters:

op - The operation to this dictionary

v - The values to use on the left hand side.

sparseSafe - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary

colIndexes - The column indexes to consider inside v.

left - A Boolean specifying if the operation is done on the left or right side of the dictionary.

Returns:

A new dictionary containing the updated values.

applyBinaryRowOpLeft

public abstract ADictionary applyBinaryRowOpLeft(BinaryOperator op,
                                                 double[] v,
                                                 boolean sparseSafe,
                                                 int[] colIndexes)

Apply binary row operation on this dictionary on the left side.

Parameters:

op - The operation to this dictionary

v - The values to use on the left hand side.

sparseSafe - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary

colIndexes - The column indexes to consider inside v.

Returns:

A new dictionary containing the updated values.

applyBinaryRowOpRight

public abstract ADictionary applyBinaryRowOpRight(BinaryOperator op,
                                                  double[] v,
                                                  boolean sparseSafe,
                                                  int[] colIndexes)

Apply binary row operation on this dictionary on the right side.

Parameters:

op - The operation to this dictionary

v - The values to use on the right hand side.

sparseSafe - boolean specifying if the operation is safe, and therefore dont need to allocate an extended dictionary

colIndexes - The column indexes to consider inside v.

Returns:

A new dictionary containing the updated values.

clone

public abstract ADictionary clone()

Returns a deep clone of the dictionary.

Overrides:

clone in class Object

cloneAndExtend

public abstract ADictionary cloneAndExtend(int len)

Clone the dictionary, and extend size of the dictionary by a given length

Parameters:

len - The length to extend the dictionary, it is assumed this value is positive.

Returns:

a clone of the dictionary, extended by len.

aggregateCols

public abstract void aggregateCols(double[] c,
                                   Builtin fn,
                                   int[] colIndexes)

Aggregates the columns into the target double array provided.

Parameters:

c - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.

fn - The function to apply to individual columns

colIndexes - The mapping to the target columns from the individual columns

write

public abstract void write(DataOutput out)
                    throws IOException

Write the dictionary to a DataOutput.

Parameters:

out - the output sink to write the dictionary to.

Throws:

IOException - if the sink fails.

getExactSizeOnDisk

public abstract long getExactSizeOnDisk()

Calculate the space consumption if the dictionary is stored on disk.

Returns:

the long count of bytes to store the dictionary.

isLossy

public abstract boolean isLossy()

Specify if the Dictionary is lossy.

Returns:

A boolean

getNumberOfValues

public abstract int getNumberOfValues(int ncol)

Get the number of distinct tuples given that the column group has n columns

Parameters:

ncol - The number of Columns in the ColumnGroup.

Returns:

the number of value tuples contained in the dictionary.

sumAllRowsToDouble

public abstract double[] sumAllRowsToDouble(boolean square,
                                            int nrColumns)

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Parameters:

square - If each entry should be squared.

nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

a double array containing the row sums from this dictionary.

sumRow

public abstract double sumRow(int k,
                              boolean square,
                              int nrColumns)

Sum the values at a specific row.

Parameters:

k - The row index to sum

square - If each entry should be squared.

nrColumns - The number of columns

Returns:

The sum of the row.

colSum

public abstract double[] colSum(int[] counts,
                                int nCol)

get the column sum of this dictionary only.

Parameters:

counts - the counts of the values contained

nCol - The number of columns contained in each tuple.

Returns:

the colSums of this column group.

colSum

public abstract void colSum(double[] c,
                            int[] counts,
                            int[] colIndexes,
                            boolean square)

Get the column sum of the values contained in the dictionary

Parameters:

c - The output array allocated to contain all column groups output.

counts - The counts of the individual tuples.

colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

square - Specify if the values should be squared

sum

public abstract double sum(int[] counts,
                           int nCol)

Get the sum of the values contained in the dictionary

Parameters:

counts - The counts of the individual tuples

nCol - The number of columns contained

Returns:

The sum scaled by the counts provided.

sumsq

public abstract double sumsq(int[] counts,
                             int nCol)

Get the square sum of the values contained in the dictionary

Parameters:

counts - The counts of the individual tuples

nCol - The number of columns contained

Returns:

The square sum scaled by the counts provided.

getString

public abstract String getString(int colIndexes)

Get a string representation of the dictionary, that considers the layout of the data.

Parameters:

colIndexes - The number of columns in the dictionary.

Returns:

A string that is nicer to print.

addMaxAndMin

public abstract void addMaxAndMin(double[] ret,
                                  int[] colIndexes)

This method adds the max and min values contained in the dictionary to corresponding cells in the ret variable. One use case for this method is the squash operation, to go from an overlapping state to normal compression.

Parameters:

ret - The double array that contains all columns min and max.

colIndexes - The column indexes contained in this dictionary.

sliceOutColumnRange

public abstract ADictionary sliceOutColumnRange(int idxStart,
                                                int idxEnd,
                                                int previousNumberOfColumns)

Modify the dictionary by removing columns not within the index range.

Parameters:

idxStart - The column index to start at.

idxEnd - The column index to end at (not inclusive)

previousNumberOfColumns - The number of columns contained in the dictionary.

Returns:

A dictionary containing the sliced out columns values only.

reExpandColumns

public abstract ADictionary reExpandColumns(int max)

return a new Dictionary that have re expanded all values, based on the entries already contained.

Parameters:

max - The number of output columns possible.

Returns:

The re expanded Dictionary.

containsValue

public abstract boolean containsValue(double pattern)

Detect if the dictionary contains a specific value.

Parameters:

pattern - The value to search for

Returns:

true if the value is contained else false.

getNumberNonZeros

public abstract long getNumberNonZeros(int[] counts,
                                       int nCol)

Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.

Parameters:

counts - The counts of each dictionary entry

nCol - The number of columns in this dictionary

Returns:

The nonZero count

addToEntry

public abstract void addToEntry(Dictionary d,
                                int fr,
                                int to,
                                int nCol)

Copies and adds the dictionary entry from this dictionary to the d dictionary

Parameters:

d - the target dictionary

fr - the from index

to - the to index

nCol - the number of columns

getMostCommonTuple

public double[] getMostCommonTuple(int[] counts,
                                   int nCol)

Get the most common tuple element contained in the dictionary returns null if that tuple is all zero values.

Parameters:

counts - The counts of the individual tuples contained, managed by the column group.

nCol - The number of columns contained in this dictionary

Returns:

a new double array containing the most common value

getTuple

public abstract double[] getTuple(int index,
                                  int nCol)

Get the values contained in a specific tuple of the dictionary. If the entire row is zero return null.

Parameters:

index - The index where the values are located

nCol - The number of columns contained in this dictionary

Returns:

a materialized double array containing the tuple.

subtractTuple

public abstract ADictionary subtractTuple(double[] tuple)

Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.

Parameters:

tuple - a double list representing a tuple, it is given that the tuple with is the same as this dictionaries.

Returns:

a new instance of dictionary with the tuple subtracted.

getAsMatrixBlockDictionary

public abstract MatrixBlockDictionary getAsMatrixBlockDictionary(int nCol)

Get this dictionary as a matrixBlock dictionary. This allows us to use optimized kernels coded elsewhere in the system, such as matrix multiplication.

Parameters:

nCol - The number of columns contained in this column group.

Returns:

A Dictionary containing a MatrixBlock.

scaleTuples

public abstract ADictionary scaleTuples(int[] scaling,
                                        int nCol)

Scale all tuples contained in the dictionary by the scaling factor given in the int list.

Parameters:

scaling - The ammout to multiply the given tuples with

nCol - The number of columns contained in this column group.

Returns:

A New dictionary (since we don't want to modify the underlying dictionary)

preaggValuesFromDense

public abstract ADictionary preaggValuesFromDense(int numVals,
                                                  int[] colIndexes,
                                                  int[] aggregateColumns,
                                                  double[] b,
                                                  int cut)

Pre Aggregate values for right Matrix Multiplication.

Parameters:

numVals - The number of values contained in this dictionary

colIndexes - The column indexes that is associated with the parent column group

aggregateColumns - The column to aggregate, this is preprocessed, to find remove consideration for empty columns

b - The values in the right hand side matrix

cut - The number of columns in b.

Returns:

A new dictionary with the pre aggregated values.

replace

public abstract ADictionary replace(double pattern,
                                    double replace,
                                    int nCol)

Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index.

Parameters:

pattern - The value to look for

replace - The value to replace the other value with

nCol - The number of columns contained in the dictionary.

Returns:

A new Column Group, reusing the index structure but with new values.

replaceZeroAndExtend

public abstract ADictionary replaceZeroAndExtend(double replace,
                                                 int nCol)

product

public abstract double product(int[] counts,
                               int nCol)

colProduct

public abstract void colProduct(double[] res,
                                int[] counts,
                                int[] colIndexes)