Package org.apache.sysds.runtime.compress.colgroup.dictionary

Class IdentityDictionarySlice

java.lang.Object

org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

org.apache.sysds.runtime.compress.colgroup.dictionary.IdentityDictionary

org.apache.sysds.runtime.compress.colgroup.dictionary.IdentityDictionarySlice

All Implemented Interfaces:

Serializable

public class IdentityDictionarySlice
extends IdentityDictionary

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

ADictionary.DictType

Constructor Summary

Constructors

Constructor	Description
IdentityDictionarySlice(int nRowCol, int l, int u)	Create a Identity matrix dictionary slice.

Method Summary

Modifier and Type	Method	Description
void	aggregateCols(double[] c, Builtin fn, IColIndex colIndexes)	Aggregates the columns into the target double array provided.
double[]	aggregateRows(Builtin fn, int nCol)	Aggregate all entries in the rows.
ADictionary	clone()	Returns a deep clone of the dictionary.
void	colProduct(double[] res, int[] counts, IColIndex colIndexes)	Calculate the column product of the dictionary weighted by counts.
void	colSum(double[] c, int[] counts, IColIndex colIndexes)	Get the column sum of the values contained in the dictionary
void	colSumSq(double[] c, int[] counts, IColIndex colIndexes)	Get the column sum of the values contained in the dictionary
boolean	containsValue(double pattern)	Detect if the dictionary contains a specific value.
boolean	equals(ADictionary o)
ADictionary.DictType	getDictType()	Get the dictionary type this dictionary is.
long	getExactSizeOnDisk()	Calculate the space consumption if the dictionary is stored on disk.
long	getInMemorySize()	Returns the memory usage of the dictionary.
static long	getInMemorySize(int numberColumns)
MatrixBlockDictionary	getMBDict()
MatrixBlockDictionary	getMBDict(int nCol)	Get this dictionary as a MatrixBlock dictionary.
long	getNumberNonZeros(int[] counts, int nCol)	Calculate the number of non zeros in the dictionary.
int	getNumberOfValues(int ncol)	Get the number of distinct tuples given that the column group has n columns
double	getSparsity()	Get the sparsity of the dictionary.
String	getString(int colIndexes)	Get a string representation of the dictionary, that considers the layout of the data.
double	getValue(int i)	Get Specific value contained in the dictionary at index.
double	getValue(int r, int c, int nCol)	Get Specific value contain in dictionary at index.
double[]	getValues()	Get all the values contained in the dictionary as a linearized double array.
double[]	productAllRowsToDouble(int nCol)	Method to product all rows to a column vector.
double[]	productAllRowsToDoubleWithDefault(double[] defaultTuple)	Method to product all rows to a column vector with a default value added in the end.
static IdentityDictionary	read(DataInput in)
ADictionary	replace(double pattern, double replace, int nCol)	Make a copy of the values, and replace all values that match pattern with replacement value.
ADictionary	replaceWithReference(double pattern, double replace, double[] reference)	Make a copy of the values, and replace all values that match pattern with replacement value.
ADictionary	scaleTuples(int[] scaling, int nCol)	Scale all tuples contained in the dictionary by the scaling factor given in the int list.
ADictionary	sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns)	Modify the dictionary by removing columns not within the index range.
double	sum(int[] counts, int ncol)	Get the sum of the values contained in the dictionary
double[]	sumAllRowsToDouble(int nrColumns)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
double[]	sumAllRowsToDoubleSq(int nrColumns)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
double[]	sumAllRowsToDoubleWithDefault(double[] defaultTuple)	Do exactly the same as the sumAllRowsToDouble but also sum the array given to a extra index in the end of the array.
double[]	sumAllRowsToDoubleWithReference(double[] reference)	Method used as a pre-aggregate of each tuple in the dictionary, to single double values with a reference.
double	sumSq(int[] counts, int ncol)	Get the square sum of the values contained in the dictionary
String	toString()
void	write(DataOutput out)	Write the dictionary to a DataOutput.

Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.IdentityDictionary

addToEntry, addToEntry, addToEntryVectorized, aggregate, aggregateColsWithReference, aggregateRowsWithDefault, aggregateRowsWithReference, aggregateWithReference, applyScalarOp, applyScalarOpAndAppend, applyScalarOpWithReference, applyUnaryOp, applyUnaryOpAndAppend, applyUnaryOpWithReference, binOpLeft, binOpLeftAndAppend, binOpLeftWithReference, binOpRight, binOpRight, binOpRightAndAppend, binOpRightWithReference, centralMoment, centralMomentWithDefault, centralMomentWithReference, colProductWithReference, colSumSqWithReference, containsValueWithReference, getNumberNonZerosWithReference, multiplyScalar, preaggValuesFromDense, product, productAllRowsToDoubleWithReference, productWithDefault, productWithReference, rexpandCols, rexpandColsWithReference, subtractTuple, sumAllRowsToDoubleSqWithDefault, sumAllRowsToDoubleSqWithReference, sumSqWithReference

Methods inherited from class org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary

centralMoment, centralMomentWithDefault, centralMomentWithReference, equals

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

IdentityDictionarySlice

public IdentityDictionarySlice(int nRowCol,
                               int l,
                               int u)

Create a Identity matrix dictionary slice. It behaves as if allocated a Sparse Matrix block but exploits that the structure is known to have certain properties.

Parameters:

nRowCol - the number of rows and columns in this identity matrix.

l - the index lower to start at

u - the index upper to end at (not inclusive)

Method Detail

getValues

public double[] getValues()

Description copied from class: ADictionary

Get all the values contained in the dictionary as a linearized double array.

Overrides:

getValues in class IdentityDictionary

Returns:

linearized double array

getValue

public double getValue(int i)

Description copied from class: ADictionary

Get Specific value contained in the dictionary at index.

Overrides:

getValue in class IdentityDictionary

Parameters:

i - The index to extract the value from

Returns:

The value contained at the index

getValue

public final double getValue(int r,
                             int c,
                             int nCol)

Description copied from class: ADictionary

Get Specific value contain in dictionary at index.

Overrides:

getValue in class IdentityDictionary

Parameters:

r - Row target

c - Col target

nCol - nCol in dictionary

Returns:

value

getInMemorySize

public long getInMemorySize()

Description copied from class: ADictionary

Returns the memory usage of the dictionary.

Overrides:

getInMemorySize in class IdentityDictionary

Returns:

a long value in number of bytes for the dictionary.

getInMemorySize

public static long getInMemorySize(int numberColumns)

aggregateRows

public double[] aggregateRows(Builtin fn,
                              int nCol)

Description copied from class: ADictionary

Aggregate all entries in the rows.

Overrides:

aggregateRows in class IdentityDictionary

Parameters:

fn - The aggregate function

nCol - The number of columns contained in the dictionary.

Returns:

Aggregates for this dictionary tuples.

aggregateCols

public void aggregateCols(double[] c,
                          Builtin fn,
                          IColIndex colIndexes)

Description copied from class: ADictionary

Aggregates the columns into the target double array provided.

Overrides:

aggregateCols in class IdentityDictionary

Parameters:

c - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.

fn - The function to apply to individual columns

colIndexes - The mapping to the target columns from the individual columns

clone

public ADictionary clone()

Description copied from class: ADictionary

Returns a deep clone of the dictionary.

Overrides:

clone in class IdentityDictionary

getDictType

public ADictionary.DictType getDictType()

Description copied from class: ADictionary

Get the dictionary type this dictionary is.

Overrides:

getDictType in class IdentityDictionary

Returns:

The Dictionary type this is.

getNumberOfValues

public int getNumberOfValues(int ncol)

Description copied from class: ADictionary

Get the number of distinct tuples given that the column group has n columns

Overrides:

getNumberOfValues in class IdentityDictionary

Parameters:

ncol - The number of Columns in the ColumnGroup.

Returns:

the number of value tuples contained in the dictionary.

sumAllRowsToDouble

public double[] sumAllRowsToDouble(int nrColumns)

Description copied from class: ADictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Overrides:

sumAllRowsToDouble in class IdentityDictionary

Parameters:

nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleWithDefault

public double[] sumAllRowsToDoubleWithDefault(double[] defaultTuple)

Description copied from class: ADictionary

Do exactly the same as the sumAllRowsToDouble but also sum the array given to a extra index in the end of the array.

Overrides:

sumAllRowsToDoubleWithDefault in class IdentityDictionary

Parameters:

defaultTuple - The default row to sum in the end index returned.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleWithReference

public double[] sumAllRowsToDoubleWithReference(double[] reference)

Description copied from class: ADictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values with a reference.

Overrides:

sumAllRowsToDoubleWithReference in class IdentityDictionary

Parameters:

reference - The reference values to add to each cell.

Returns:

a double array containing the row sums from this dictionary.

sumAllRowsToDoubleSq

public double[] sumAllRowsToDoubleSq(int nrColumns)

Description copied from class: ADictionary

Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.

Overrides:

sumAllRowsToDoubleSq in class IdentityDictionary

Parameters:

nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

a double array containing the row sums from this dictionary.

productAllRowsToDouble

public double[] productAllRowsToDouble(int nCol)

Description copied from class: ADictionary

Method to product all rows to a column vector.

Overrides:

productAllRowsToDouble in class IdentityDictionary

Parameters:

nCol - The number of columns in the ColGroup to know how to get the values from the dictionary.

Returns:

A row product

productAllRowsToDoubleWithDefault

public double[] productAllRowsToDoubleWithDefault(double[] defaultTuple)

Description copied from class: ADictionary

Method to product all rows to a column vector with a default value added in the end.

Overrides:

productAllRowsToDoubleWithDefault in class IdentityDictionary

Parameters:

defaultTuple - The default row that aggregate to last cell

Returns:

A row product

colSum

public void colSum(double[] c,
                   int[] counts,
                   IColIndex colIndexes)

Description copied from class: ADictionary

Get the column sum of the values contained in the dictionary

Overrides:

colSum in class IdentityDictionary

Parameters:

c - The output array allocated to contain all column groups output.

counts - The counts of the individual tuples.

colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

colSumSq

public void colSumSq(double[] c,
                     int[] counts,
                     IColIndex colIndexes)

Description copied from class: ADictionary

Get the column sum of the values contained in the dictionary

Overrides:

colSumSq in class IdentityDictionary

Parameters:

c - The output array allocated to contain all column groups output.

counts - The counts of the individual tuples.

colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

colProduct

public void colProduct(double[] res,
                       int[] counts,
                       IColIndex colIndexes)

Description copied from class: ADictionary

Calculate the column product of the dictionary weighted by counts.

Overrides:

colProduct in class IdentityDictionary

Parameters:

res - The result vector to put the result into

counts - The weighted count of individual tuples

colIndexes - The column indexes.

sum

public double sum(int[] counts,
                  int ncol)

Description copied from class: ADictionary

Get the sum of the values contained in the dictionary

Overrides:

sum in class IdentityDictionary

Parameters:

counts - The counts of the individual tuples

ncol - The number of columns contained

Returns:

The sum scaled by the counts provided.

sumSq

public double sumSq(int[] counts,
                    int ncol)

Description copied from class: ADictionary

Get the square sum of the values contained in the dictionary

Overrides:

sumSq in class IdentityDictionary

Parameters:

counts - The counts of the individual tuples

ncol - The number of columns contained

Returns:

The square sum scaled by the counts provided.

sliceOutColumnRange

public ADictionary sliceOutColumnRange(int idxStart,
                                       int idxEnd,
                                       int previousNumberOfColumns)

Description copied from class: ADictionary

Modify the dictionary by removing columns not within the index range.

Overrides:

sliceOutColumnRange in class IdentityDictionary

Parameters:

idxStart - The column index to start at.

idxEnd - The column index to end at (not inclusive)

previousNumberOfColumns - The number of columns contained in the dictionary.

Returns:

A dictionary containing the sliced out columns values only.

containsValue

public boolean containsValue(double pattern)

Description copied from class: ADictionary

Detect if the dictionary contains a specific value.

Overrides:

containsValue in class IdentityDictionary

Parameters:

pattern - The value to search for

Returns:

true if the value is contained else false.

getNumberNonZeros

public long getNumberNonZeros(int[] counts,
                              int nCol)

Description copied from class: ADictionary

Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.

Overrides:

getNumberNonZeros in class IdentityDictionary

Parameters:

counts - The counts of each dictionary entry

nCol - The number of columns in this dictionary

Returns:

The nonZero count

getMBDict

public MatrixBlockDictionary getMBDict()

Overrides:

getMBDict in class IdentityDictionary

getMBDict

public MatrixBlockDictionary getMBDict(int nCol)

Description copied from class: ADictionary

Get this dictionary as a MatrixBlock dictionary. This allows us to use optimized kernels coded elsewhere in the system, such as matrix multiplication. Return null if the matrix is empty.

Overrides:

getMBDict in class IdentityDictionary

Parameters:

nCol - The number of columns contained in this column group.

Returns:

A Dictionary containing a MatrixBlock.

getString

public String getString(int colIndexes)

Description copied from class: ADictionary

Get a string representation of the dictionary, that considers the layout of the data.

Overrides:

getString in class IdentityDictionary

Parameters:

colIndexes - The number of columns in the dictionary.

Returns:

A string that is nicer to print.

toString

public String toString()

Overrides:

toString in class IdentityDictionary

scaleTuples

public ADictionary scaleTuples(int[] scaling,
                               int nCol)

Description copied from class: ADictionary

Scale all tuples contained in the dictionary by the scaling factor given in the int list.

Overrides:

scaleTuples in class IdentityDictionary

Parameters:

scaling - The amount to multiply the given tuples with

nCol - The number of columns contained in this column group.

Returns:

A New dictionary (since we don't want to modify the underlying dictionary)

write

public void write(DataOutput out)
           throws IOException

Description copied from class: ADictionary

Write the dictionary to a DataOutput.

Overrides:

write in class IdentityDictionary

Parameters:

out - the output sink to write the dictionary to.

Throws:

IOException - if the sink fails.

read

public static IdentityDictionary read(DataInput in)
                               throws IOException

Throws:

IOException

getExactSizeOnDisk

public long getExactSizeOnDisk()

Description copied from class: ADictionary

Calculate the space consumption if the dictionary is stored on disk.

Overrides:

getExactSizeOnDisk in class IdentityDictionary

Returns:

the long count of bytes to store the dictionary.

replace

public ADictionary replace(double pattern,
                           double replace,
                           int nCol)

Description copied from class: ADictionary

Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index.

Overrides:

replace in class IdentityDictionary

Parameters:

pattern - The value to look for

replace - The value to replace the other value with

nCol - The number of columns contained in the dictionary.

Returns:

A new Column Group, reusing the index structure but with new values.

replaceWithReference

public ADictionary replaceWithReference(double pattern,
                                        double replace,
                                        double[] reference)

Description copied from class: ADictionary

Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index. With reference such that each value in the dict is considered offset by the values contained in the reference.

Overrides:

replaceWithReference in class IdentityDictionary

Parameters:

pattern - The value to look for

replace - The value to replace the other value with

reference - The reference tuple to add to all entries when replacing

Returns:

A new Column Group, reusing the index structure but with new values.

getSparsity

public double getSparsity()

Description copied from class: ADictionary

Get the sparsity of the dictionary.

Overrides:

getSparsity in class IdentityDictionary

Returns:

a sparsity between 0 and 1

equals

public boolean equals(ADictionary o)

Overrides:

equals in class IdentityDictionary