Package org.apache.sysds.runtime.frame.data.columns

Class DDCArray<T>

java.lang.Object

org.apache.sysds.runtime.frame.data.columns.Array<T>

org.apache.sysds.runtime.frame.data.columns.ACompressedArray<T>

org.apache.sysds.runtime.frame.data.columns.DDCArray<T>

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class DDCArray<T>
extends ACompressedArray<T>

A dense dictionary version of an column array

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.sysds.runtime.frame.data.columns.Array

Array.ArrayIterator

Constructor Summary

Constructors

Constructor	Description
DDCArray(Array<T> dict, AMapToData map)

Method Summary

Modifier and Type	Method	Description
Pair<Types.ValueType,Boolean>	analyzeValueType(int maxCells)	Analyze the column to figure out if the value type can be refined to a better type.
Array<T>	append(Array<T> other)	Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this.
Array<?>	changeTypeWithNulls(Types.ValueType t)
Array<T>	clone()	Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays
static <T> Array<T>	compressToDDC(Array<T> arr)	Try to compress array into DDC format.
static <T> Array<?>	compressToDDC(Array<T> arr, Types.ValueType vt, boolean containsNull)	compress and change value.
boolean	containsNull()	analyze if the array contains null values.
boolean	equals(Array<T> other)
static long	estimateInMemorySize(int memSizeBitPerElement, int estDistinct, int nRow)
double[]	extractDouble(double[] ret, int rl, int ru)
T	get(int index)	Get the value at a given index.
byte[]	getAsByteArray()	Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.
double	getAsDouble(int i)	Get the index's value.
double	getAsNaNDouble(int i)	Get the index's value.
Array<T>	getDict()
long	getExactSerializedSize()	Get the exact serialized size on disk of this array.
ArrayFactory.FrameArrayType	getFrameArrayType()	Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.
long	getInMemorySize()	Get in memory size, not counting reference to this object.
AMapToData	getMap()
Types.ValueType	getValueType()	Get the current value type of this array.
double	hashDouble(int idx)	Hash the given index of the array.
boolean	isEmpty()	Get if this array is empty, aka filled with empty values.
boolean	isNotEmpty(int i)
boolean	isShallowSerialize()	analyze if this array can be shallow serialized.
DDCArray<T>	nullDict()
boolean	possiblyContainsNaN()
static DDCArray<?>	read(DataInput in)
void	readFields(DataInput in)
Array<T>	select(boolean[] select, int nTrue)	Slice out the true indices in the select input and return the sub array.
Array<T>	select(int[] indices)	Slice out the specified indices and return the sub array.
<J> DDCArray<J>	setDict(Array<J> dict)
Array<T>	slice(int rl, int ru)	Slice out the sub range and return new array with the specified type.
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.sysds.runtime.frame.data.columns.ACompressedArray

append, append, fill, fill, get, reset, set, set, set, set, set, setFromOtherType, setFromOtherTypeNz, setNz, statistics

Methods inherited from class org.apache.sysds.runtime.frame.data.columns.Array

analyzeValueType, baseMemoryCost, changeType, changeType, createMapping, equals, findEmpty, findEmptyInverse, getCache, getIterator, getMinMaxLength, getNulls, getRecodeMap, safeChangeType, setCache, setFromOtherTypeNz, setNz, size

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DDCArray

public DDCArray(Array<T> dict,
                AMapToData map)

Method Detail

getDict

public Array<T> getDict()

getMap

public AMapToData getMap()

setDict

public <J> DDCArray<J> setDict(Array<J> dict)

nullDict

public DDCArray<T> nullDict()

compressToDDC

public static <T> Array<T> compressToDDC(Array<T> arr)

Try to compress array into DDC format.

Type Parameters:

T - The type of the Array

Parameters:

arr - The array to try to compress

Returns:

Either a compressed version or the original.

compressToDDC

public static <T> Array<?> compressToDDC(Array<T> arr,
                                         Types.ValueType vt,
                                         boolean containsNull)

compress and change value.

Type Parameters:

T - The type of the array.

Parameters:

arr - The array to compress

vt - The value type to target

containsNull - If the array contains null.

Returns:

a compressed column group.

write

public void write(DataOutput out)
           throws IOException

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Throws:

IOException

read

public static DDCArray<?> read(DataInput in)
                        throws IOException

Throws:

IOException

get

public T get(int index)

Description copied from class: Array

Get the value at a given index. This method returns objects that have a high overhead in allocation. Therefore it is not as efficient as using the vectorized operations specified in the object.

Specified by:

get in class Array<T>

Parameters:

index - The index to query

Returns:

The value returned as an object

extractDouble

public double[] extractDouble(double[] ret,
                              int rl,
                              int ru)

Overrides:

extractDouble in class Array<T>

getAsDouble

public double getAsDouble(int i)

Description copied from class: Array

Get the index's value. returns 0 in case of Null.

Specified by:

getAsDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

getAsNaNDouble

public double getAsNaNDouble(int i)

Description copied from class: Array

Get the index's value. returns Double.NaN in case of Null.

Overrides:

getAsNaNDouble in class Array<T>

Parameters:

i - index to get value from

Returns:

the value

append

public Array<T> append(Array<T> other)

Description copied from class: Array

Append other array, if the other array is fitting in current allocated size use that allocated size, otherwise allocate new array to combine the other with this. This method should use the set range function, and should be preferred over the append single values.

Specified by:

append in class Array<T>

Parameters:

other - The other array of same type to append to this.

Returns:

The combined arrays.

slice

public Array<T> slice(int rl,
                      int ru)

Description copied from class: Array

Slice out the sub range and return new array with the specified type. If the conversion fails fallback to normal slice.

Specified by:

slice in class Array<T>

Parameters:

rl - row start

ru - row end (not included)

Returns:

A new array of sub range.

getAsByteArray

public byte[] getAsByteArray()

Description copied from class: Array

Return the current allocated Array as a byte[], this is used to serialize the allocated Arrays out to the PythonAPI.

Specified by:

getAsByteArray in class Array<T>

Returns:

The array as bytes

getValueType

public Types.ValueType getValueType()

Description copied from class: Array

Get the current value type of this array.

Specified by:

getValueType in class Array<T>

Returns:

The current value type.

analyzeValueType

public Pair<Types.ValueType,Boolean> analyzeValueType(int maxCells)

Description copied from class: Array

Analyze the column to figure out if the value type can be refined to a better type. The return is in two parts, first the type it can be, second if it contains nulls.

Specified by:

analyzeValueType in class Array<T>

Parameters:

maxCells - maximum number of cells to analyze

Returns:

A better or equivalent value type to represent the column, including null information.

getFrameArrayType

public ArrayFactory.FrameArrayType getFrameArrayType()

Description copied from class: Array

Get the internal FrameArrayType, to specify the encoding of the Types, note there are more Frame Array Types than there is ValueTypes.

Specified by:

getFrameArrayType in class Array<T>

Returns:

The FrameArrayType

getExactSerializedSize

public long getExactSerializedSize()

Description copied from class: Array

Get the exact serialized size on disk of this array.

Specified by:

getExactSerializedSize in class Array<T>

Returns:

The exact size on disk

changeTypeWithNulls

public Array<?> changeTypeWithNulls(Types.ValueType t)

Overrides:

changeTypeWithNulls in class Array<T>

isShallowSerialize

public boolean isShallowSerialize()

Description copied from class: Array

analyze if this array can be shallow serialized. to allow caching without modification.

Specified by:

isShallowSerialize in class Array<T>

Returns:

boolean saying true if shallow serialization is available

isEmpty

public boolean isEmpty()

Description copied from class: Array

Get if this array is empty, aka filled with empty values.

Specified by:

isEmpty in class Array<T>

Returns:

boolean saying true if empty

select

public Array<T> select(int[] indices)

Description copied from class: Array

Slice out the specified indices and return the sub array.

Specified by:

select in class Array<T>

Parameters:

indices - The indices to slice out

Returns:

the sliced out indices in an array format

select

public Array<T> select(boolean[] select,
                       int nTrue)

Description copied from class: Array

Slice out the true indices in the select input and return the sub array.

Specified by:

select in class Array<T>

Parameters:

select - a boolean vector specifying what to select

nTrue - number of true values inside select

Returns:

the sliced out indices in an array format

isNotEmpty

public boolean isNotEmpty(int i)

Specified by:

isNotEmpty in class Array<T>

clone

public Array<T> clone()

Description copied from class: Array

Overwrite of the java internal clone function for arrays, return a clone of underlying data that is mutable, (not immutable data.) Immutable data is dependent on the individual allocated arrays

Specified by:

clone in class Array<T>

Returns:

A clone

hashDouble

public double hashDouble(int idx)

Description copied from class: Array

Hash the given index of the array. It is allowed to return NaN on null elements.

Specified by:

hashDouble in class Array<T>

Parameters:

idx - The index to hash

Returns:

The hash value of that index.

getInMemorySize

public long getInMemorySize()

Description copied from class: Array

Get in memory size, not counting reference to this object.

Overrides:

getInMemorySize in class Array<T>

Returns:

the size in memory of this object.

estimateInMemorySize

public static long estimateInMemorySize(int memSizeBitPerElement,
                                        int estDistinct,
                                        int nRow)

containsNull

public boolean containsNull()

Description copied from class: Array

analyze if the array contains null values.

Overrides:

containsNull in class Array<T>

Returns:

If the array contains null.

equals

public boolean equals(Array<T> other)

Specified by:

equals in class Array<T>

possiblyContainsNaN

public boolean possiblyContainsNaN()

Specified by:

possiblyContainsNaN in class Array<T>

toString

public String toString()

Overrides:

toString in class Array<T>