Package org.apache.sysds.runtime.frame.data.columns

Class DDCArray<T>

java.lang.Object

org.apache.sysds.runtime.frame.data.columns.Array<T>

org.apache.sysds.runtime.frame.data.columns.ACompressedArray<T>

org.apache.sysds.runtime.frame.data.columns.DDCArray<T>

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class DDCArray<T>
extends ACompressedArray<T>

A dense dictionary version of an column array

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.frame.data.columns.Array

Array.ArrayIterator

Field Summary

Fields inherited from class org.apache.sysds.runtime.frame.data.columns.Array

ROW_PARALLELIZATION_THRESHOLD

Constructor Summary

Constructors

Constructor	Description
DDCArray(Array<T> dict, AMapToData map)

Method Summary

Modifier and Type	Method	Description
Pair<Types.ValueType,Boolean>	analyzeValueType(int maxCells)	Analyze the column to figure out if the value type can be refined to a better type.
Array<T>	append(Array<T> other)	Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this.
Array<?>	changeType(Types.ValueType t)	Change the allocated array to a different type.
Array<?>	changeTypeWithNulls(Types.ValueType t)
Array<T>	clone()	Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays
static <T> Array<T>	compressToDDC(Array<T> arr)
static <T> Array<T>	compressToDDC(Array<T> arr, int estimateUnique)	Try to compress array into DDC format.
boolean	containsNull()	analyze if the array contains null values.
boolean	equals(Array<T> other)	Equals operation on arrays.
static long	estimateInMemorySize(int memSizeBitPerElement, int estDistinct, int nRow)
double[]	extractDouble(double[] ret, int rl, int ru)	Extract the sub array into the ret array as doubles.
T	get(int index)	Get the value at a given index.
byte[]	getAsByteArray()	Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.
double	getAsDouble(int i)	Get the index's value.
double	getAsNaNDouble(int i)	Get the index's value.
Array<T>	getDict()
long	getExactSerializedSize()	Get the exact serialized size on disk of this array.
ArrayFactory.FrameArrayType	getFrameArrayType()	Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.
long	getInMemorySize()	Get in memory size, not counting reference to this object.
T	getInternal(int index)	Get the internal value at a given index.
AMapToData	getMap()
Types.ValueType	getValueType()	Get the current value type of this array.
double	hashDouble(int idx)	Hash the given index of the array.
boolean	isEmpty()	Get if this array is empty, aka filled with empty values.
boolean	isNotEmpty(int i)
boolean	isShallowSerialize()	analyze if this array can be shallow serialized.
double[]	minMax()	Get the minimum and maximum double value of this array.
double[]	minMax(int l, int u)	Get the minimum and maximum double value of a specific sub part of this array.
DDCArray<T>	nullDict()
boolean	possiblyContainsNaN()
static DDCArray<?>	read(DataInput in)
void	readFields(DataInput in)
Array<T>	select(boolean[] select, int nTrue)	Slice out the true indices in the select input and return the sub array.
Array<T>	select(int[] indices)	Slice out the specified indices and return the sub array.
void	set(int rl, int ru, Array<T> value, int rlSrc)	Set range to given arrays value with an offset into other array
<J> DDCArray<J>	setDict(Array<J> dict)
DDCArray<T>	setMap(AMapToData map)
Array<T>	slice(int rl, int ru)	Slice out the sub range and return new array with the specified type.
ArrayCompressionStatistics	statistics(int nSamples)	Get the compression statistics of this array allocation.
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.sysds.runtime.frame.data.columns.ACompressedArray

append, append, fill, fill, get, reset, set, set, set, setFromOtherType, setFromOtherTypeNz, setNz

Methods inherited from class org.apache.sysds.runtime.frame.data.columns.Array

analyzeValueType, baseMemoryCost, changeType, changeType, changeType, changeTypeWithNulls, changeTypeWithNulls, equals, findEmpty, findEmptyInverse, getCache, getIterator, getMinMaxLength, getNulls, getRecodeMap, getRecodeMap, getRecodeMap, set, setCache, setFromOtherTypeNz, setM, setM, setNz, size

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DDCArray

public DDCArray(Array<T> dict,
                AMapToData map)

Method Detail

getDict

public Array<T> getDict()

getMap

public AMapToData getMap()

setDict

public <J> DDCArray<J> setDict(Array<J> dict)

setMap

public DDCArray<T> setMap(AMapToData map)

nullDict

public DDCArray<T> nullDict()

compressToDDC

public static <T> Array<T> compressToDDC(Array<T> arr)

compressToDDC

public static <T> Array<T> compressToDDC(Array<T> arr,
                                         int estimateUnique)

Try to compress array into DDC format.

Type Parameters:

T - The type of the Array

Parameters:

arr - The array to try to compress

estimateUnique - The estimated number of unique values

Returns:

Either a compressed version or the original.

write

public void write(DataOutput out)
           throws IOException

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Throws:

IOException

read

public static DDCArray<?> read(DataInput in)
                        throws IOException

Throws:

IOException

get

public T get(int index)

Description copied from class: Array

Get the value at a given index. This method returns objects that have a high overhead in allocation. Therefore it is not as efficient as using the vectorized operations specified in the object.

Specified by:

get in class Array<T>

Parameters:

index - The index to query

Returns:

The value returned as an object

getInternal

public T getInternal(int index)

Description copied from class: Array

Get the internal value at a given index. For instance HashIntegerArray would return the underlying long not a string.

Overrides:

getInternal in class Array<T>

Parameters:

index - the index to get

Returns:

The value to get

extractDouble

public double[] extractDouble(double[] ret,
                              int rl,
                              int ru)

Description copied from class: Array

Extract the sub array into the ret array as doubles. The ret array is filled from - rl, meaning that the ret array should be of length ru - rl.

Overrides:

extractDouble in class Array<T>

Parameters:

ret - The array to return

rl - The row to start at

ru - The row to end at (not inclusive.)

Returns:

The ret array given as argument

getAsDouble

public double getAsDouble(int i)

Description copied from class: Array

Get the index's value. returns 0 in case of Null.

Specified by:

getAsDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

getAsNaNDouble

public double getAsNaNDouble(int i)

Description copied from class: Array

Get the index's value. returns Double.NaN in case of Null.

Overrides:

getAsNaNDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

append

public Array<T> append(Array<T> other)

Description copied from class: Array

Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this. This method should use the set range function, and should be preferred over the append single values.

Specified by:

append in class Array<T>

Parameters:

other - The other array of same type to append to this.

Returns:

The combined arrays.

slice

public Array<T> slice(int rl,
                      int ru)

Description copied from class: Array

Slice out the sub range and return new array with the specified type. If the conversion fails fallback to normal slice.

Specified by:

slice in class Array<T>

Parameters:

rl - row start

ru - row end (not included)

Returns:

A new array of sub range.

getAsByteArray

public byte[] getAsByteArray()

Description copied from class: Array

Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.

Specified by:

getAsByteArray in class Array<T>

Returns:

The array as bytes

getValueType

public Types.ValueType getValueType()

Description copied from class: Array

Get the current value type of this array.

Specified by:

getValueType in class Array<T>

Returns:

The current value type.

analyzeValueType

public Pair<Types.ValueType,Boolean> analyzeValueType(int maxCells)

Description copied from class: Array

Analyze the column to figure out if the value type can be refined to a better type. The return is in two parts, first the type it can be, second if it contains nulls.

Specified by:

analyzeValueType in class Array<T>

Parameters:

maxCells - maximum number of cells to analyze

Returns:

A better or equivalent value type to represent the column, including null information.

set

public void set(int rl,
                int ru,
                Array<T> value,
                int rlSrc)

Description copied from class: Array

Set range to given arrays value with an offset into other array

Overrides:

set in class Array<T>

Parameters:

rl - row lower

ru - row upper (inclusive)

value - value array to take values from

rlSrc - the offset into the value array to take values from

getFrameArrayType

public ArrayFactory.FrameArrayType getFrameArrayType()

Description copied from class: Array

Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.

Specified by:

getFrameArrayType in class Array<T>

Returns:

The FrameArrayType

getExactSerializedSize

public long getExactSerializedSize()

Description copied from class: Array

Get the exact serialized size on disk of this array.

Specified by:

getExactSerializedSize in class Array<T>

Returns:

The exact size on disk

changeType

public Array<?> changeType(Types.ValueType t)

Description copied from class: Array

Change the allocated array to a different type. If the type is the same a deep copy is returned for safety.

Specified by:

changeType in class ACompressedArray<T>

Parameters:

t - The type to change to

Returns:

A new column array.

changeTypeWithNulls

public Array<?> changeTypeWithNulls(Types.ValueType t)

Overrides:

changeTypeWithNulls in class Array<T>

isShallowSerialize

public boolean isShallowSerialize()

Description copied from class: Array

analyze if this array can be shallow serialized. to allow caching without modification.

Specified by:

isShallowSerialize in class Array<T>

Returns:

boolean saying true if shallow serialization is available

isEmpty

public boolean isEmpty()

Description copied from class: Array

Get if this array is empty, aka filled with empty values.

Specified by:

isEmpty in class Array<T>

Returns:

boolean saying true if empty

select

public Array<T> select(int[] indices)

Description copied from class: Array

Slice out the specified indices and return the sub array.

Specified by:

select in class Array<T>

Parameters:

indices - The indices to slice out

Returns:

the sliced out indices in an array format

select

public Array<T> select(boolean[] select,
                       int nTrue)

Description copied from class: Array

Slice out the true indices in the select input and return the sub array.

Specified by:

select in class Array<T>

Parameters:

select - a boolean vector specifying what to select

nTrue - number of true values inside select

Returns:

the sliced out indices in an array format

isNotEmpty

public boolean isNotEmpty(int i)

Specified by:

isNotEmpty in class Array<T>

clone

public Array<T> clone()

Description copied from class: Array

Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays

Specified by:

clone in class Array<T>

Returns:

A clone

hashDouble

public double hashDouble(int idx)

Description copied from class: Array

Hash the given index of the array. It is allowed to return NaN on null elements.

Specified by:

hashDouble in class Array<T>

Parameters:

idx - The index to hash

Returns:

The hash value of that index.

getInMemorySize

public long getInMemorySize()

Description copied from class: Array

Get in memory size, not counting reference to this object.

Overrides:

getInMemorySize in class Array<T>

Returns:

the size in memory of this object.

estimateInMemorySize

public static long estimateInMemorySize(int memSizeBitPerElement,
                                        int estDistinct,
                                        int nRow)

containsNull

public boolean containsNull()

Description copied from class: Array

analyze if the array contains null values.

Overrides:

containsNull in class Array<T>

Returns:

If the array contains null.

equals

public boolean equals(Array<T> other)

Description copied from class: Array

Equals operation on arrays.

Specified by:

equals in class Array<T>

Parameters:

other - The other array to compare to.

Returns:

True if the arrays are equivalent.

possiblyContainsNaN

public boolean possiblyContainsNaN()

Specified by:

possiblyContainsNaN in class Array<T>

minMax

public double[] minMax()

Description copied from class: Array

Get the minimum and maximum double value of this array. Note that we ignore NaN Values.

Overrides:

minMax in class Array<T>

Returns:

The min and max in index 0 and 1 of the array.

minMax

public double[] minMax(int l,
                       int u)

Description copied from class: Array

Get the minimum and maximum double value of a specific sub part of this array. Note that we ignore NaN Values.

Overrides:

minMax in class Array<T>

Parameters:

l - The lower index to search from

u - The upper index to end at (not inclusive)

Returns:

The min and max in index 0 and 1 of the array in the range.

statistics

public ArrayCompressionStatistics statistics(int nSamples)

Description copied from class: Array

Get the compression statistics of this array allocation.

Specified by:

statistics in class ACompressedArray<T>

Parameters:

nSamples - The number of sample elements suggested (not forced) to be used.

Returns:

The compression statistics of this array.

toString

public String toString()

Overrides:

toString in class Array<T>