Package org.apache.sysds.runtime.compress.colgroup.dictionary

Class IdentityDictionary

java.lang.Object

org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

org.apache.sysds.runtime.compress.colgroup.dictionary.ACachingMBDictionary

org.apache.sysds.runtime.compress.colgroup.dictionary.AIdentityDictionary

org.apache.sysds.runtime.compress.colgroup.dictionary.IdentityDictionary

All Implemented Interfaces:

Serializable, IDictionary

public class IdentityDictionary
extends AIdentityDictionary

A specialized dictionary that exploits the fact that the contained dictionary is an Identity Matrix.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.sysds.runtime.compress.colgroup.dictionary.IDictionary

IDictionary.DictType

Field Summary

Fields inherited from interface org.apache.sysds.runtime.compress.colgroup.dictionary.IDictionary

LOG

Method Summary

Modifier and Type	Method	Description
void	addToEntry(double[] v, int fr, int to, int nCol)	Adds the dictionary entry from this dictionary to the d dictionary
void	addToEntry(double[] v, int fr, int to, int nCol, int rep)	Adds the dictionary entry from this dictionary to the v dictionary rep times.
void	addToEntryVectorized(double[] v, int f1, int f2, int f3, int f4, int f5, int f6, int f7, int f8, int t1, int t2, int t3, int t4, int t5, int t6, int t7, int t8, int nCol)	Vectorized add to entry, this call helps with a bit of locality for the cache.
double	aggregate(double init, Builtin fn)	Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.
void	aggregateCols(double[] c, Builtin fn, IColIndex colIndexes)	Aggregates the columns into the target double array provided.
double[]	aggregateRows(Builtin fn, int nCol)	Aggregate all entries in the rows.
IDictionary	binOpRight(BinaryOperator op, double[] v, IColIndex colIndexes)	Apply binary row operation on the right side.
IDictionary	clone()	Returns a deep clone of the dictionary.
void	colProduct(double[] res, int[] counts, IColIndex colIndexes)	Calculate the column product of the dictionary weighted by counts.
void	colSum(double[] c, int[] counts, IColIndex colIndexes)	Get the column sum of the values contained in the dictionary
void	colSumSq(double[] c, int[] counts, IColIndex colIndexes)	Get the column sum of the values contained in the dictionary
int[]	countNNZZeroColumns(int[] counts)	Count the number of non zero values in each column of the dictionary, multiplied with the counts
static IDictionary	create(int nRowCol)	Create an identity matrix dictionary.
static IDictionary	create(int nRowCol, boolean withEmpty)	Create an identity matrix dictionary, It behaves as if allocated a Sparse Matrix block but exploits that the structure is known to have certain properties.
MatrixBlockDictionary	createMBDict(int nCol)
boolean	equals(IDictionary o)	Indicate if the other dictionary is equal to this.
IDictionary.DictType	getDictType()	Get the dictionary type this dictionary is.
long	getExactSizeOnDisk()	Calculate the space consumption if the dictionary is stored on disk.
long	getInMemorySize()	Returns the memory usage of the dictionary.
static long	getInMemorySize(int numberColumns)
MatrixBlockDictionary	getMBDict()
long	getNumberNonZeros(int[] counts, int nCol)	Calculate the number of non zeros in the dictionary.
int	getNumberOfColumns(int nrow)	Get the number of columns in this dictionary, provided you know the number of values, or rows.
int	getNumberOfValues(int ncol)	Get the number of distinct tuples given that the column group has n columns
double	getSparsity()	Get the sparsity of the dictionary.
String	getString(int colIndexes)	Get a string representation of the dictionary, that considers the layout of the data.
double	getValue(int i)	Get Specific value contained in the dictionary at index.
double	getValue(int r, int c, int nCol)	Get Specific value contain in dictionary at index.
double[]	getValues()	Get all the values contained in the dictionary as a linearized double array.
void	MMDictDense(double[] left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)	Matrix multiplication of dictionaries left side dense and transposed.
void	MMDictScalingDense(double[] left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result, int[] scaling)	Matrix multiplication of dictionaries left side dense and transposed.
void	multiplyScalar(double v, double[] ret, int off, int dictIdx, IColIndex cols)	Multiply the v value with the dictionary entry at dictIdx and add it to the ret matrix at the columns specified in the int array.
IDictionary	preaggValuesFromDense(int numVals, IColIndex colIndexes, IColIndex aggregateColumns, double[] b, int cut)	Pre Aggregate values for Right Matrix Multiplication.
static IdentityDictionary	read(DataInput in)
IDictionary	sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns)	Modify the dictionary by removing columns not within the index range.
double	sum(int[] counts, int ncol)	Get the sum of the values contained in the dictionary
double[]	sumAllRowsToDouble(int nrColumns)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
double[]	sumAllRowsToDoubleSq(int nrColumns)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
double[]	sumAllRowsToDoubleWithDefault(double[] defaultTuple)	Do exactly the same as the sumAllRowsToDouble but also sum the array given to a extra index in the end of the array.
double[]	sumAllRowsToDoubleWithReference(double[] reference)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values with a reference.
double	sumSq(int[] counts, int ncol)	Get the square sum of the values contained in the dictionary
String	toString()
void	write(DataOutput out)	Write the dictionary to a DataOutput.

Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.AIdentityDictionary

containsValue, productAllRowsToDouble, productAllRowsToDoubleWithDefault, withEmpty

Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.ACachingMBDictionary

getMBDict

Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

aggregateColsWithReference, aggregateRowsWithDefault, aggregateRowsWithReference, aggregateWithReference, append, applyScalarOp, applyScalarOpAndAppend, applyScalarOpWithReference, applyUnaryOp, applyUnaryOpAndAppend, applyUnaryOpWithReference, binOpLeft, binOpLeftAndAppend, binOpLeftWithReference, binOpRight, binOpRightAndAppend, binOpRightWithReference, cbind, centralMoment, centralMoment, centralMomentWithDefault, centralMomentWithDefault, centralMomentWithReference, centralMomentWithReference, colProductWithReference, colSumSqWithReference, containsValueWithReference, correctNan, equals, equals, getNumberNonZerosWithReference, getRow, MMDict, MMDictScaling, MMDictScalingSparse, MMDictSparse, product, productAllRowsToDoubleWithReference, productWithDefault, productWithReference, putDense, putSparse, reorder, replace, replaceWithReference, rexpandCols, rexpandColsWithReference, rightMMPreAggSparse, scaleTuples, subtractTuple, sumAllRowsToDoubleSqWithDefault, sumAllRowsToDoubleSqWithReference, sumSqWithReference, TSMMToUpperTriangle, TSMMToUpperTriangleDense, TSMMToUpperTriangleDenseScaling, TSMMToUpperTriangleScaling, TSMMToUpperTriangleSparse, TSMMToUpperTriangleSparseScaling, TSMMWithScaling

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

create

public static IDictionary create(int nRowCol)

Create an identity matrix dictionary. It behaves as if allocated a Sparse Matrix block but exploits that the structure is known to have certain properties.

Parameters:

nRowCol - The number of rows and columns in this identity matrix.

Returns:

a Dictionary instance.

create

public static IDictionary create(int nRowCol,
                                 boolean withEmpty)

Create an identity matrix dictionary, It behaves as if allocated a Sparse Matrix block but exploits that the structure is known to have certain properties.

Parameters:

nRowCol - The number of rows and columns in this identity matrix.

withEmpty - If the matrix should contain an empty row in the end.

Returns:

a Dictionary instance.

getValues

public double[] getValues()

Description copied from interface: IDictionary

Get all the values contained in the dictionary as a linearized double array.

Specified by:

getValues in interface IDictionary

Overrides:

getValues in class ADictionary

Returns:

linearized double array

getValue

public double getValue(int i)

Description copied from interface: IDictionary

Get Specific value contained in the dictionary at index.

Specified by:

getValue in interface IDictionary

Overrides:

getValue in class ADictionary

Parameters:

i - The index to extract the value from

Returns:

The value contained at the index

getValue

public double getValue(int r,
                       int c,
                       int nCol)

Description copied from interface: IDictionary

Get Specific value contain in dictionary at index.

Specified by:

getValue in interface IDictionary

Overrides:

getValue in class ADictionary

Parameters:

r - Row target

c - Col target

nCol - nCol in dictionary

Returns:

value

getInMemorySize

public long getInMemorySize()

Description copied from interface: IDictionary

Returns the memory usage of the dictionary.

Returns:

a long value in number of bytes for the dictionary.

getInMemorySize

public static long getInMemorySize(int numberColumns)

aggregate

public double aggregate(double init,
                        Builtin fn)

Description copied from interface: IDictionary

Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.

Specified by:

aggregate in interface IDictionary

Overrides:

aggregate in class ADictionary

Parameters:

init - The initial Value, in cases such as Max value, this could be -infinity

fn - The Function to apply to values

Returns:

The aggregated value as a double.

aggregateRows

public double[] aggregateRows(Builtin fn,
                              int nCol)

Description copied from interface: IDictionary

Aggregate all entries in the rows.

Specified by:

aggregateRows in interface IDictionary

Overrides:

aggregateRows in class ADictionary

Parameters:

fn - The aggregate function

nCol - The number of columns contained in the dictionary.

Returns:

Aggregates for this dictionary tuples.

aggregateCols

public void aggregateCols(double[] c,
                          Builtin fn,
                          IColIndex colIndexes)

Description copied from interface: IDictionary

Aggregates the columns into the target double array provided.

Specified by:

aggregateCols in interface IDictionary

Overrides:

aggregateCols in class ADictionary

Parameters:

c - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.

fn - The function to apply to individual columns

colIndexes - The mapping to the target columns from the individual columns

binOpRight

public IDictionary binOpRight(BinaryOperator op,
                              double[] v,
                              IColIndex colIndexes)

Description copied from interface: IDictionary

Apply binary row operation on the right side.

Specified by:

binOpRight in interface IDictionary

Overrides:

binOpRight in class ADictionary

Parameters:

op - The operation to this dictionary

v - The values to use on the right hand side.

colIndexes - The column indexes to consider inside v.

Returns:

A new dictionary containing the updated values.

clone

public IDictionary clone()

Description copied from interface: IDictionary

Returns a deep clone of the dictionary.

Specified by:

clone in interface IDictionary

Specified by:

clone in class ADictionary

Returns:

A deep clone

getDictType

public IDictionary.DictType getDictType()

Description copied from interface: IDictionary

Get the dictionary type this dictionary is.

Returns:

The Dictionary type this is.

getNumberOfValues

public int getNumberOfValues(int ncol)

Description copied from interface: IDictionary

Get the number of distinct tuples given that the column group has n columns

Parameters:

ncol - The number of Columns in the ColumnGroup.

Returns:

the number of value tuples contained in the dictionary.

getNumberOfColumns

public int getNumberOfColumns(int nrow)

Description copied from interface: IDictionary

Get the number of columns in this dictionary, provided you know the number of values, or rows.

Parameters:

nrow - The number of rows/values known inside this dictionary

Returns:

The number of columns

sumAllRowsToDouble

public double[] sumAllRowsToDouble(int nrColumns)

Description copied from interface: IDictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Specified by:

sumAllRowsToDouble in interface IDictionary

Overrides:

sumAllRowsToDouble in class ADictionary

Parameters:

nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleWithDefault

public double[] sumAllRowsToDoubleWithDefault(double[] defaultTuple)

Description copied from interface: IDictionary

Do exactly the same as the sumAllRowsToDouble but also sum the array given to a extra index in the end of the array.

Specified by:

sumAllRowsToDoubleWithDefault in interface IDictionary

Overrides:

sumAllRowsToDoubleWithDefault in class ADictionary

Parameters:

defaultTuple - The default row to sum in the end index returned.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleWithReference

public double[] sumAllRowsToDoubleWithReference(double[] reference)

Description copied from interface: IDictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values with a reference.

Specified by:

sumAllRowsToDoubleWithReference in interface IDictionary

Overrides:

sumAllRowsToDoubleWithReference in class ADictionary

Parameters:

reference - The reference values to add to each cell.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleSq

public double[] sumAllRowsToDoubleSq(int nrColumns)

Description copied from interface: IDictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Specified by:

sumAllRowsToDoubleSq in interface IDictionary

Overrides:

sumAllRowsToDoubleSq in class ADictionary

Parameters:

nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

a double array containing the row sums from this dictionary.

colSum

public void colSum(double[] c,
                   int[] counts,
                   IColIndex colIndexes)

Description copied from interface: IDictionary

Get the column sum of the values contained in the dictionary

Specified by:

colSum in interface IDictionary

Overrides:

colSum in class ADictionary

Parameters:

c - The output array allocated to contain all column groups output.

counts - The counts of the individual tuples.

colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

colSumSq

public void colSumSq(double[] c,
                     int[] counts,
                     IColIndex colIndexes)

Description copied from interface: IDictionary

Get the column sum of the values contained in the dictionary

Specified by:

colSumSq in interface IDictionary

Overrides:

colSumSq in class ADictionary

Parameters:

c - The output array allocated to contain all column groups output.

counts - The counts of the individual tuples.

colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

colProduct

public void colProduct(double[] res,
                       int[] counts,
                       IColIndex colIndexes)

Description copied from interface: IDictionary

Calculate the column product of the dictionary weighted by counts.

Specified by:

colProduct in interface IDictionary

Overrides:

colProduct in class ADictionary

Parameters:

res - The result vector to put the result into

counts - The weighted count of individual tuples

colIndexes - The column indexes.

sum

public double sum(int[] counts,
                  int ncol)

Description copied from interface: IDictionary

Get the sum of the values contained in the dictionary

Specified by:

sum in interface IDictionary

Overrides:

sum in class ADictionary

Parameters:

counts - The counts of the individual tuples

ncol - The number of columns contained

Returns:

The sum scaled by the counts provided.

sumSq

public double sumSq(int[] counts,
                    int ncol)

Description copied from interface: IDictionary

Get the square sum of the values contained in the dictionary

Specified by:

sumSq in interface IDictionary

Overrides:

sumSq in class ADictionary

Parameters:

counts - The counts of the individual tuples

ncol - The number of columns contained

Returns:

The square sum scaled by the counts provided.

sliceOutColumnRange

public IDictionary sliceOutColumnRange(int idxStart,
                                       int idxEnd,
                                       int previousNumberOfColumns)

Description copied from interface: IDictionary

Modify the dictionary by removing columns not within the index range.

Specified by:

sliceOutColumnRange in interface IDictionary

Overrides:

sliceOutColumnRange in class ADictionary

Parameters:

idxStart - The column index to start at.

idxEnd - The column index to end at (not inclusive)

previousNumberOfColumns - The number of columns contained in the dictionary.

Returns:

A dictionary containing the sliced out columns values only.

getNumberNonZeros

public long getNumberNonZeros(int[] counts,
                              int nCol)

Description copied from interface: IDictionary

Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.

Parameters:

counts - The counts of each dictionary entry

nCol - The number of columns in this dictionary

Returns:

The nonZero count

countNNZZeroColumns

public int[] countNNZZeroColumns(int[] counts)

Description copied from interface: IDictionary

Count the number of non zero values in each column of the dictionary, multiplied with the counts

Specified by:

countNNZZeroColumns in interface IDictionary

Overrides:

countNNZZeroColumns in class ADictionary

Parameters:

counts - The counts to multiply with.

Returns:

The nonzero count of each column in the dictionary.

addToEntry

public final void addToEntry(double[] v,
                             int fr,
                             int to,
                             int nCol)

Description copied from interface: IDictionary

Adds the dictionary entry from this dictionary to the d dictionary

Specified by:

addToEntry in interface IDictionary

Overrides:

addToEntry in class ADictionary

Parameters:

v - The target dictionary (dense double array)

fr - The from index is the tuple index to copy from.

to - The to index is the row index to copy into.

nCol - The number of columns in both cases

addToEntry

public void addToEntry(double[] v,
                       int fr,
                       int to,
                       int nCol,
                       int rep)

Description copied from interface: IDictionary

Adds the dictionary entry from this dictionary to the v dictionary rep times.

Specified by:

addToEntry in interface IDictionary

Overrides:

addToEntry in class ADictionary

Parameters:

v - The target dictionary (dense double array)

fr - The from index is the tuple index to copy from.

to - The to index is the row index to copy into.

nCol - The number of columns in both cases

rep - The number of repetitions to apply (simply multiply do not loop)

addToEntryVectorized

public void addToEntryVectorized(double[] v,
                                 int f1,
                                 int f2,
                                 int f3,
                                 int f4,
                                 int f5,
                                 int f6,
                                 int f7,
                                 int f8,
                                 int t1,
                                 int t2,
                                 int t3,
                                 int t4,
                                 int t5,
                                 int t6,
                                 int t7,
                                 int t8,
                                 int nCol)

Description copied from interface: IDictionary

Vectorized add to entry, this call helps with a bit of locality for the cache.

Specified by:

addToEntryVectorized in interface IDictionary

Overrides:

addToEntryVectorized in class ADictionary

Parameters:

v - The target dictionary (dense double array)

f1 - From index 1

f2 - From index 2

f3 - From index 3

f4 - From index 4

f5 - From index 5

f6 - From index 6

f7 - From index 7

f8 - From index 8

t1 - To index 1

t2 - To index 2

t3 - To index 3

t4 - To index 4

t5 - To index 5

t6 - To index 6

t7 - To index 7

t8 - To index 8

nCol - Number of columns in the dictionary

getMBDict

public MatrixBlockDictionary getMBDict()

Overrides:

getMBDict in class ADictionary

createMBDict

public MatrixBlockDictionary createMBDict(int nCol)

Specified by:

createMBDict in class ACachingMBDictionary

write

public void write(DataOutput out)
           throws IOException

Description copied from interface: IDictionary

Write the dictionary to a DataOutput.

Parameters:

out - the output sink to write the dictionary to.

Throws:

IOException - if the sink fails.

read

public static IdentityDictionary read(DataInput in)
                               throws IOException

Throws:

IOException

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from interface: IDictionary

Calculate the space consumption if the dictionary is stored on disk.

Returns:

the long count of bytes to store the dictionary.

preaggValuesFromDense

public IDictionary preaggValuesFromDense(int numVals,
                                         IColIndex colIndexes,
                                         IColIndex aggregateColumns,
                                         double[] b,
                                         int cut)

Description copied from interface: IDictionary

Pre Aggregate values for Right Matrix Multiplication.

Specified by:

preaggValuesFromDense in interface IDictionary

Overrides:

preaggValuesFromDense in class ADictionary

Parameters:

numVals - The number of values contained in this dictionary

colIndexes - The column indexes that is associated with the parent column group

aggregateColumns - The column to aggregate, this is preprocessed, to find remove consideration for empty columns

b - The values in the right hand side matrix

cut - The number of columns in b.

Returns:

A new dictionary with the pre aggregated values.

getSparsity

public double getSparsity()

Description copied from interface: IDictionary

Get the sparsity of the dictionary.

Specified by:

getSparsity in interface IDictionary

Overrides:

getSparsity in class ADictionary

Returns:

a sparsity between 0 and 1

multiplyScalar

public void multiplyScalar(double v,
                           double[] ret,
                           int off,
                           int dictIdx,
                           IColIndex cols)

Description copied from interface: IDictionary

Multiply the v value with the dictionary entry at dictIdx and add it to the ret matrix at the columns specified in the int array.

Specified by:

multiplyScalar in interface IDictionary

Overrides:

multiplyScalar in class ADictionary

Parameters:

v - Value to multiply

ret - Output dense double array location

off - Offset into the ret array that the "row" output starts at

dictIdx - The dictionary entry to multiply.

cols - The columns to multiply into of the output.

MMDictDense

public void MMDictDense(double[] left,
                        IColIndex rowsLeft,
                        IColIndex colsRight,
                        MatrixBlock result)

Description copied from interface: IDictionary

Matrix multiplication of dictionaries left side dense and transposed. The right side is this.

Specified by:

MMDictDense in interface IDictionary

Overrides:

MMDictDense in class ADictionary

Parameters:

left - Dense left side (treat as if it is transposed but it is physically not)

rowsLeft - Offset rows on the left

colsRight - Offset cols on the right

result - The output matrix block

MMDictScalingDense

public void MMDictScalingDense(double[] left,
                               IColIndex rowsLeft,
                               IColIndex colsRight,
                               MatrixBlock result,
                               int[] scaling)

Description copied from interface: IDictionary

Matrix multiplication of dictionaries left side dense and transposed. The Right side is this, the scaling factor is used to multiply each element with.

Specified by:

MMDictScalingDense in interface IDictionary

Overrides:

MMDictScalingDense in class ADictionary

Parameters:

left - Dense left side (Dense dictionary)

rowsLeft - Offset rows on the left (That dictionaries column indexes)

colsRight - Offset cols on the right (This dictionaries column indexes)

result - The output matrix block, guaranteed to be allocated as dense.

scaling - The scaling factor to multiply each entry with.

equals

public boolean equals(IDictionary o)

Description copied from interface: IDictionary

Indicate if the other dictionary is equal to this.

Parameters:

o - The other object

Returns:

If it is equal

getString

public String getString(int colIndexes)

Description copied from interface: IDictionary

Get a string representation of the dictionary, that considers the layout of the data.

Parameters:

colIndexes - The number of columns in the dictionary.

Returns:

A string that is nicer to print.

toString

public String toString()

Overrides:

toString in class Object