| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
Connection.convertToFrame(InputStream input,
int rows,
int cols)
Converts an input stream of a string frame in textcell format
into a frame block.
|
FrameBlock |
Connection.convertToFrame(InputStream input,
int rows,
int cols,
String format)
Converts an input stream of a frame in csv or textcell format
into a frame block.
|
FrameBlock |
Connection.convertToFrame(InputStream input,
String meta)
Converts an input stream of a string frame in csv or textcell format
into a frame block.
|
FrameBlock |
Connection.convertToFrame(String input,
int rows,
int cols)
Converts an input string representation of a frame in textcell format
into a frame block.
|
FrameBlock |
Connection.convertToFrame(String input,
String meta)
Converts an input string representation of a frame in csv or textcell format
into a frame block.
|
FrameBlock |
ResultVariables.getFrameBlock(String varname)
Obtain the frame represented by the given output variable.
|
FrameBlock |
Connection.readTransformMetaDataFromFile(String metapath)
Reads transform meta data from an HDFS file path and converts it into an in-memory
FrameBlock object.
|
FrameBlock |
Connection.readTransformMetaDataFromFile(String spec,
String metapath)
Reads transform meta data from an HDFS file path and converts it into an in-memory
FrameBlock object.
|
FrameBlock |
Connection.readTransformMetaDataFromFile(String spec,
String metapath,
String colDelim)
Reads transform meta data from an HDFS file path and converts it into an in-memory
FrameBlock object.
|
FrameBlock |
Connection.readTransformMetaDataFromPath(String metapath)
Reads transform meta data from the class path and converts it into an in-memory
FrameBlock object.
|
FrameBlock |
Connection.readTransformMetaDataFromPath(String spec,
String metapath)
Reads transform meta data from the class path and converts it into an in-memory
FrameBlock object.
|
FrameBlock |
Connection.readTransformMetaDataFromPath(String spec,
String metapath,
String colDelim)
Reads transform meta data from the class path and converts it into an in-memory
FrameBlock object.
|
| Modifier and Type | Method and Description |
|---|---|
void |
PreparedScript.setFrame(String varname,
FrameBlock frame,
boolean reuse)
Binds a frame object to a registered input variable.
|
| Modifier and Type | Method and Description |
|---|---|
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
MLContextConversionUtil.dataFrameToFrameBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame,
FrameMetadata frameMetadata)
Convert a
DataFrame to a JavaPairRDD<Long, FrameBlock>
binary-block frame. |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
MLContextConversionUtil.frameObjectToBinaryBlocks(FrameObject frameObject,
SparkExecutionContext sparkExecutionContext)
Convert a
FrameObject to a JavaPairRDD<Long, FrameBlock>. |
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
Frame.toBinaryBlocks()
Obtain the frame as a
JavaPairRDD<Long, FrameBlock> |
| Modifier and Type | Method and Description |
|---|---|
static FrameObject |
MLContextConversionUtil.frameBlockToFrameObject(String variableName,
FrameBlock frameBlock,
FrameMetadata frameMetadata)
Convert a
FrameBlock to a FrameObject. |
| Modifier and Type | Method and Description |
|---|---|
static FrameObject |
MLContextConversionUtil.binaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks)
Convert a
JavaPairRDD<Long, FrameBlock> to a FrameObject. |
static FrameObject |
MLContextConversionUtil.binaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks,
FrameMetadata frameMetadata)
Convert a
JavaPairRDD<Long, FrameBlock> to a FrameObject. |
| Constructor and Description |
|---|
Frame(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks,
FrameMetadata frameMetadata)
Create a Frame, specifying the SystemDS binary-block frame and its
metadata.
|
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
ExecutionContext.getFrameInput(String varName)
Pins a frame variable into memory and returns the internal frame block.
|
static FrameBlock |
SparkExecutionContext.toFrameBlock(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> rdd,
Types.ValueType[] schema,
int rlen,
int clen) |
static FrameBlock |
SparkExecutionContext.toFrameBlock(RDDObject rdd,
Types.ValueType[] schema,
int rlen,
int clen) |
| Modifier and Type | Method and Description |
|---|---|
PartitionedBroadcast<FrameBlock> |
SparkExecutionContext.getBroadcastForFrameVariable(String varname) |
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
SparkExecutionContext.getFrameBinaryBlockRDDHandleForVariable(String varname)
Spark instructions should call this for all frame inputs except broadcast
variables.
|
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
SparkExecutionContext.toFrameJavaPairRDD(org.apache.spark.api.java.JavaSparkContext sc,
FrameBlock src) |
| Modifier and Type | Method and Description |
|---|---|
static FrameObject |
ExecutionContext.createFrameObject(FrameBlock fb) |
void |
ExecutionContext.setFrameOutput(String varName,
FrameBlock outputData) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
SparkExecutionContext.toFrameJavaPairRDD(org.apache.spark.api.java.JavaSparkContext sc,
FrameBlock src) |
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
SparkExecutionContext.toFrameBlock(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> rdd,
Types.ValueType[] schema,
int rlen,
int clen) |
| Constructor and Description |
|---|
DecodeMatrix(long input,
long outputID,
FrameBlock meta,
Decoder decoder) |
| Modifier and Type | Method and Description |
|---|---|
scala.Tuple2<Long,FrameBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformApplyFunction.call(scala.Tuple2<Long,FrameBlock> in) |
scala.Tuple2<Long,FrameBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformDecodeFunction.call(scala.Tuple2<MatrixIndexes,MatrixBlock> in) |
| Modifier and Type | Method and Description |
|---|---|
Iterator<scala.Tuple2<Integer,Object>> |
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuildFunction.call(Iterator<scala.Tuple2<Long,FrameBlock>> iter) |
Iterator<scala.Tuple2<Integer,FrameBlock.ColumnMetadata>> |
MultiReturnParameterizedBuiltinSPInstruction.TransformEncodeBuild2Function.call(Iterator<scala.Tuple2<Long,FrameBlock>> iter) |
scala.Tuple2<Long,FrameBlock> |
ParameterizedBuiltinSPInstruction.RDDTransformApplyFunction.call(scala.Tuple2<Long,FrameBlock> in) |
scala.Tuple2<Long,Long> |
ParameterizedBuiltinSPInstruction.RDDTransformApplyOffsetFunction.call(scala.Tuple2<Long,FrameBlock> in) |
| Modifier and Type | Method and Description |
|---|---|
void |
FrameReblockBuffer.flushBufferToBinaryBlocks(ArrayList<Pair<Long,FrameBlock>> outList) |
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
CopyFrameBlockFunction.call(FrameBlock arg0) |
| Modifier and Type | Method and Description |
|---|---|
scala.Tuple2<Long,FrameBlock> |
CopyFrameBlockPairFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
CopyFrameBlockFunction.call(FrameBlock arg0) |
| Modifier and Type | Method and Description |
|---|---|
Boolean |
IsFrameBlockInRange.call(scala.Tuple2<Long,FrameBlock> kv) |
Iterator<String> |
ConvertFrameBlockToIJVLines.call(scala.Tuple2<Long,FrameBlock> kv) |
scala.Tuple2<Long,FrameBlock> |
CopyFrameBlockPairFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
| Modifier and Type | Method and Description |
|---|---|
scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.LongFrameToLongWritableFrameFunction.call(scala.Tuple2<Long,FrameBlock> arg0) |
scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.LongWritableToSerFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
scala.Tuple2<Long,FrameBlock> |
FrameRDDConverterUtils.LongWritableFrameToLongFrameFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> input,
DataCharacteristics mc,
Types.ValueType[] schema,
boolean hasHeader,
String delim,
boolean fill,
double fillValue,
Set<String> naStrings) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.csvToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaRDD<String> input,
DataCharacteristics mcOut,
Types.ValueType[] schema,
boolean hasHeader,
String delim,
boolean fill,
double fillValue,
Set<String> naStrings) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
DataCharacteristics mc,
boolean containsID) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.dataFrameToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df,
DataCharacteristics mc,
boolean containsID,
Pair<String[],Types.ValueType[]> out) |
static List<scala.Tuple2<Long,FrameBlock>> |
SparkUtils.fromIndexedFrameBlock(List<Pair<Long,FrameBlock>> in) |
static scala.Tuple2<Long,FrameBlock> |
SparkUtils.fromIndexedFrameBlock(Pair<Long,FrameBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
DataCharacteristics mcIn) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.matrixBlockToBinaryBlockLongIndex(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> input,
DataCharacteristics dcIn) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.textCellToBinaryBlock(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> in,
DataCharacteristics mcOut,
Types.ValueType[] schema) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDConverterUtils.textCellToBinaryBlockLongIndex(org.apache.spark.api.java.JavaSparkContext sc,
org.apache.spark.api.java.JavaPairRDD<Long,org.apache.hadoop.io.Text> input,
DataCharacteristics mc,
Types.ValueType[] schema) |
static Pair<Long,FrameBlock> |
SparkUtils.toIndexedFrameBlock(scala.Tuple2<Long,FrameBlock> in) |
| Modifier and Type | Method and Description |
|---|---|
static org.apache.spark.api.java.JavaRDD<String> |
FrameRDDConverterUtils.binaryBlockToCsv(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
DataCharacteristics mcIn,
FileFormatPropertiesCSV props,
boolean strict) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
FrameRDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SparkSession sparkSession,
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
DataCharacteristics mc,
Types.ValueType[] schema) |
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> |
FrameRDDConverterUtils.binaryBlockToDataFrame(org.apache.spark.sql.SQLContext sqlContext,
org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in,
DataCharacteristics mc,
Types.ValueType[] schema)
Deprecated.
|
static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> |
FrameRDDConverterUtils.binaryBlockToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> input,
DataCharacteristics mcIn,
DataCharacteristics mcOut) |
static org.apache.spark.api.java.JavaRDD<String> |
FrameRDDConverterUtils.binaryBlockToTextCell(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> input,
DataCharacteristics mcIn) |
scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.LongFrameToLongWritableFrameFunction.call(scala.Tuple2<Long,FrameBlock> arg0) |
scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> |
FrameRDDConverterUtils.LongWritableToSerFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
scala.Tuple2<Long,FrameBlock> |
FrameRDDConverterUtils.LongWritableFrameToLongFrameFunction.call(scala.Tuple2<org.apache.hadoop.io.LongWritable,FrameBlock> arg0) |
static List<scala.Tuple2<Long,FrameBlock>> |
SparkUtils.fromIndexedFrameBlock(List<Pair<Long,FrameBlock>> in) |
static scala.Tuple2<Long,FrameBlock> |
SparkUtils.fromIndexedFrameBlock(Pair<Long,FrameBlock> in) |
static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> |
FrameRDDAggregateUtils.mergeByKey(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> in) |
static Pair<Long,FrameBlock> |
SparkUtils.toIndexedFrameBlock(scala.Tuple2<Long,FrameBlock> in) |
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock[] |
FrameWriter.createFrameBlocksForReuse(Types.ValueType[] schema,
String[] names,
long rlen) |
static FrameBlock |
FrameWriter.getFrameBlockForReuse(FrameBlock[] blocks) |
FrameBlock |
FrameReaderBinaryBlock.readFirstBlock(String fname)
Specific functionality of FrameReaderBinaryBlock, mostly used for testing.
|
FrameBlock |
FrameReader.readFrameFromHDFS(String fname,
long rlen,
long clen) |
FrameBlock |
FrameReader.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
long rlen,
long clen) |
FrameBlock |
FrameReaderJSONL.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
Map<String,Integer> schemaMap,
long rlen,
long clen) |
FrameBlock |
FrameReaderProto.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
abstract FrameBlock |
FrameReader.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderTextCell.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderTextCSV.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderBinaryBlock.readFrameFromHDFS(String fname,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReader.readFrameFromInputStream(InputStream is,
long rlen,
long clen) |
FrameBlock |
FrameReader.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
long rlen,
long clen) |
FrameBlock |
FrameReaderProto.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
abstract FrameBlock |
FrameReader.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderTextCell.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderTextCSV.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
FrameBlock |
FrameReaderBinaryBlock.readFrameFromInputStream(InputStream is,
Types.ValueType[] schema,
String[] names,
long rlen,
long clen) |
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
FrameWriter.getFrameBlockForReuse(FrameBlock[] blocks) |
void |
FrameWriterTextCell.writeFrameToHDFS(FrameBlock src,
String fname,
long rlen,
long clen) |
void |
FrameWriterProto.writeFrameToHDFS(FrameBlock src,
String fname,
long rlen,
long clen) |
abstract void |
FrameWriter.writeFrameToHDFS(FrameBlock src,
String fname,
long rlen,
long clen) |
void |
FrameWriterTextCSV.writeFrameToHDFS(FrameBlock src,
String fname,
long rlen,
long clen) |
void |
FrameWriterBinaryBlock.writeFrameToHDFS(FrameBlock src,
String fname,
long rlen,
long clen) |
void |
FrameWriterJSONL.writeFrameToHDFS(FrameBlock src,
String fname,
Map<String,Integer> schemaMap,
long rlen,
long clen) |
| Constructor and Description |
|---|
ReadTask(org.apache.hadoop.mapred.InputSplit split,
org.apache.hadoop.mapred.TextInputFormat informat,
org.apache.hadoop.mapred.JobConf job,
FrameBlock dest) |
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
FrameBlock.append(FrameBlock that,
FrameBlock ret,
boolean cbind)
Appends the given argument frameblock 'that' to this frameblock by
creating a deep copy to prevent side effects.
|
FrameBlock |
FrameBlock.binaryOperations(BinaryOperator bop,
FrameBlock that,
FrameBlock out)
This method performs the value comparison on two frames
if the values in both frames are equal, not equal, less than, greater than, less than/greater than and equal to
the output frame will store boolean value for each each comparison
|
FrameBlock |
FrameBlock.detectSchemaFromRow(double sampleFraction) |
FrameBlock |
FrameBlock.dropInvalidType(FrameBlock schema)
Drop the cell value which does not confirms to the data type of its column
|
FrameBlock |
FrameBlock.getColumnNamesAsFrame() |
FrameBlock |
FrameBlock.getSchemaTypeOf() |
FrameBlock |
FrameBlock.invalidByLength(MatrixBlock feaLen)
This method validates the frame data against an attribute length constrain
if data value in any cell is greater than the specified threshold of that attribute
the output frame will store a null on that cell position, thus removing the length-violating values.
|
FrameBlock |
FrameBlock.leftIndexingOperations(FrameBlock rhsFrame,
IndexRange ixrange,
FrameBlock ret) |
FrameBlock |
FrameBlock.leftIndexingOperations(FrameBlock rhsFrame,
int rl,
int ru,
int cl,
int cu,
FrameBlock ret) |
FrameBlock |
FrameBlock.map(FrameBlock.FrameMapFunction lambdaExpr) |
FrameBlock |
FrameBlock.map(String lambdaExpr) |
static FrameBlock |
FrameBlock.mergeSchema(FrameBlock temp1,
FrameBlock temp2) |
FrameBlock |
FrameBlock.slice(IndexRange ixrange,
FrameBlock ret) |
FrameBlock |
FrameBlock.slice(int rl,
int ru,
int cl,
int cu,
CacheBlock retCache)
Right indexing operations to slice a subframe out of this frame block.
|
FrameBlock |
FrameBlock.zeroOutOperations(FrameBlock result,
IndexRange range,
boolean complementary,
int iRowStartSrc,
int iRowStartDest,
int blen,
int iMaxRowsToCopy)
This function ZERO OUT the data in the slicing window applicable for this block.
|
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
FrameBlock.append(FrameBlock that,
FrameBlock ret,
boolean cbind)
Appends the given argument frameblock 'that' to this frameblock by
creating a deep copy to prevent side effects.
|
FrameBlock |
FrameBlock.binaryOperations(BinaryOperator bop,
FrameBlock that,
FrameBlock out)
This method performs the value comparison on two frames
if the values in both frames are equal, not equal, less than, greater than, less than/greater than and equal to
the output frame will store boolean value for each each comparison
|
void |
FrameBlock.copy(FrameBlock src) |
void |
FrameBlock.copy(int rl,
int ru,
int cl,
int cu,
FrameBlock src) |
FrameBlock |
FrameBlock.dropInvalidType(FrameBlock schema)
Drop the cell value which does not confirms to the data type of its column
|
FrameBlock |
FrameBlock.leftIndexingOperations(FrameBlock rhsFrame,
IndexRange ixrange,
FrameBlock ret) |
FrameBlock |
FrameBlock.leftIndexingOperations(FrameBlock rhsFrame,
int rl,
int ru,
int cl,
int cu,
FrameBlock ret) |
void |
FrameBlock.merge(FrameBlock that) |
static FrameBlock |
FrameBlock.mergeSchema(FrameBlock temp1,
FrameBlock temp2) |
static ArrayList |
OperationsOnMatrixValues.performSlice(IndexRange ixrange,
int blen,
int iix,
int jix,
FrameBlock in) |
FrameBlock |
FrameBlock.slice(IndexRange ixrange,
FrameBlock ret) |
FrameBlock |
FrameBlock.zeroOutOperations(FrameBlock result,
IndexRange range,
boolean complementary,
int iRowStartSrc,
int iRowStartDest,
int blen,
int iMaxRowsToCopy)
This function ZERO OUT the data in the slicing window applicable for this block.
|
| Modifier and Type | Method and Description |
|---|---|
static void |
OperationsOnMatrixValues.performShift(Pair<Long,FrameBlock> in,
IndexRange ixrange,
int blenLeft,
long rlen,
long clen,
ArrayList<Pair<Long,FrameBlock>> outlist) |
static void |
OperationsOnMatrixValues.performShift(Pair<Long,FrameBlock> in,
IndexRange ixrange,
int blenLeft,
long rlen,
long clen,
ArrayList<Pair<Long,FrameBlock>> outlist) |
static void |
OperationsOnMatrixValues.performSlice(Pair<Long,FrameBlock> in,
IndexRange ixrange,
int blen,
ArrayList<Pair<Long,FrameBlock>> outlist)
This function will get slice of the input frame block overlapping in overall slice(Range), slice has requested for.
|
static void |
OperationsOnMatrixValues.performSlice(Pair<Long,FrameBlock> in,
IndexRange ixrange,
int blen,
ArrayList<Pair<Long,FrameBlock>> outlist)
This function will get slice of the input frame block overlapping in overall slice(Range), slice has requested for.
|
void |
FrameBlock.slice(ArrayList<Pair<Long,FrameBlock>> outlist,
IndexRange range,
int rowCut) |
| Constructor and Description |
|---|
FrameBlock(FrameBlock that)
Copy constructor for frame blocks, which uses a shallow copy for
the schema (column types and names) but a deep copy for meta data
and actual column data.
|
| Modifier and Type | Method and Description |
|---|---|
abstract FrameBlock |
Decoder.decode(MatrixBlock in,
FrameBlock out)
Block decode API converting a matrix block into a frame block.
|
FrameBlock |
DecoderComposite.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderPassThrough.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderDummycode.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderRecode.decode(MatrixBlock in,
FrameBlock out) |
| Modifier and Type | Method and Description |
|---|---|
static Decoder |
DecoderFactory.createDecoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta) |
static Decoder |
DecoderFactory.createDecoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta,
int clen) |
static Decoder |
DecoderFactory.createDecoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta,
int minCol,
int maxCol) |
static Decoder |
DecoderFactory.createDecoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta,
int clen,
int minCol,
int maxCol) |
abstract FrameBlock |
Decoder.decode(MatrixBlock in,
FrameBlock out)
Block decode API converting a matrix block into a frame block.
|
FrameBlock |
DecoderComposite.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderPassThrough.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderDummycode.decode(MatrixBlock in,
FrameBlock out) |
FrameBlock |
DecoderRecode.decode(MatrixBlock in,
FrameBlock out) |
abstract void |
Decoder.initMetaData(FrameBlock meta) |
void |
DecoderComposite.initMetaData(FrameBlock meta) |
void |
DecoderPassThrough.initMetaData(FrameBlock meta) |
void |
DecoderDummycode.initMetaData(FrameBlock meta) |
void |
DecoderRecode.initMetaData(FrameBlock meta) |
| Modifier and Type | Method and Description |
|---|---|
FrameBlock |
EncoderRecode.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderComposite.getMetaData(FrameBlock out) |
FrameBlock |
EncoderMVImpute.getMetaData(FrameBlock out) |
FrameBlock |
EncoderOmit.getMetaData(FrameBlock out) |
abstract FrameBlock |
Encoder.getMetaData(FrameBlock out)
Construct a frame block out of the transform meta data.
|
FrameBlock |
EncoderDummycode.getMetaData(FrameBlock out) |
FrameBlock |
EncoderPassThrough.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderBin.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderFeatureHash.getMetaData(FrameBlock meta) |
| Modifier and Type | Method and Description |
|---|---|
MatrixBlock |
EncoderRecode.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderComposite.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderMVImpute.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderOmit.apply(FrameBlock in,
MatrixBlock out) |
abstract MatrixBlock |
Encoder.apply(FrameBlock in,
MatrixBlock out)
Encode input data blockwise according to existing transform meta
data (transform apply).
|
MatrixBlock |
EncoderDummycode.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderPassThrough.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderBin.apply(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderFeatureHash.apply(FrameBlock in,
MatrixBlock out) |
void |
EncoderRecode.build(FrameBlock in) |
void |
EncoderComposite.build(FrameBlock in) |
void |
EncoderMVImpute.build(FrameBlock in) |
void |
EncoderOmit.build(FrameBlock in) |
abstract void |
Encoder.build(FrameBlock in)
Build the transform meta data for the given block input.
|
void |
EncoderDummycode.build(FrameBlock in) |
void |
EncoderPassThrough.build(FrameBlock in) |
void |
EncoderBin.build(FrameBlock in) |
void |
EncoderFeatureHash.build(FrameBlock in) |
void |
EncoderRecode.buildPartial(FrameBlock in) |
static Encoder |
EncoderFactory.createEncoder(String spec,
String[] colnames,
int clen,
FrameBlock meta) |
static Encoder |
EncoderFactory.createEncoder(String spec,
String[] colnames,
int clen,
FrameBlock meta,
int minCol,
int maxCol) |
static Encoder |
EncoderFactory.createEncoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta) |
static Encoder |
EncoderFactory.createEncoder(String spec,
String[] colnames,
Types.ValueType[] schema,
FrameBlock meta,
int minCol,
int maxCol) |
static Encoder |
EncoderFactory.createEncoder(String spec,
String[] colnames,
Types.ValueType[] schema,
int clen,
FrameBlock meta) |
MatrixBlock |
EncoderRecode.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderComposite.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderMVImpute.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderOmit.encode(FrameBlock in,
MatrixBlock out) |
abstract MatrixBlock |
Encoder.encode(FrameBlock in,
MatrixBlock out)
Block encode: build and apply (transform encode).
|
MatrixBlock |
EncoderDummycode.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderPassThrough.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderBin.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderFeatureHash.encode(FrameBlock in,
MatrixBlock out) |
MatrixBlock |
EncoderComposite.getColMapping(FrameBlock meta,
MatrixBlock out) |
MatrixBlock |
Encoder.getColMapping(FrameBlock meta,
MatrixBlock out)
Obtain the column mapping of encoded frames based on the passed
meta data frame.
|
MatrixBlock |
EncoderDummycode.getColMapping(FrameBlock meta,
MatrixBlock out) |
FrameBlock |
EncoderRecode.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderComposite.getMetaData(FrameBlock out) |
FrameBlock |
EncoderMVImpute.getMetaData(FrameBlock out) |
FrameBlock |
EncoderOmit.getMetaData(FrameBlock out) |
abstract FrameBlock |
Encoder.getMetaData(FrameBlock out)
Construct a frame block out of the transform meta data.
|
FrameBlock |
EncoderDummycode.getMetaData(FrameBlock out) |
FrameBlock |
EncoderPassThrough.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderBin.getMetaData(FrameBlock meta) |
FrameBlock |
EncoderFeatureHash.getMetaData(FrameBlock meta) |
void |
EncoderRecode.initMetaData(FrameBlock meta)
Construct the recodemaps from the given input frame for all
columns registered for recode.
|
void |
EncoderComposite.initMetaData(FrameBlock out) |
void |
EncoderMVImpute.initMetaData(FrameBlock meta) |
void |
EncoderOmit.initMetaData(FrameBlock meta) |
abstract void |
Encoder.initMetaData(FrameBlock meta)
Sets up the required meta data for a subsequent call to apply.
|
void |
EncoderDummycode.initMetaData(FrameBlock meta) |
void |
EncoderPassThrough.initMetaData(FrameBlock meta) |
void |
EncoderBin.initMetaData(FrameBlock meta) |
void |
EncoderFeatureHash.initMetaData(FrameBlock meta) |
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
TfMetaUtils.readTransformMetaDataFromFile(String spec,
String metapath,
String colDelim)
Reads transform meta data from an HDFS file path and converts it into an in-memory
FrameBlock object.
|
static FrameBlock |
TfMetaUtils.readTransformMetaDataFromPath(String spec,
String metapath,
String colDelim)
Reads transform meta data from the class path and converts it into an in-memory
FrameBlock object.
|
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
DataConverter.convertToFrameBlock(MatrixBlock mb)
Converts a matrix block into a frame block of value type double.
|
static FrameBlock |
DataConverter.convertToFrameBlock(MatrixBlock mb,
Types.ValueType vt)
Converts a matrix block into a frame block of a given value type.
|
static FrameBlock |
DataConverter.convertToFrameBlock(MatrixBlock mb,
Types.ValueType[] schema) |
static FrameBlock |
DataConverter.convertToFrameBlock(String[][] data)
Converts a two dimensions string array into a frame block of
value type string.
|
static FrameBlock |
DataConverter.convertToFrameBlock(String[][] data,
Types.ValueType[] schema) |
static FrameBlock |
DataConverter.convertToFrameBlock(String[][] data,
Types.ValueType[] schema,
String[] colnames) |
static FrameBlock |
LocalFileUtils.readFrameBlockFromLocal(String fname)
Reads a frame block from local file system.
|
static FrameBlock |
LocalFileUtils.readFrameBlockFromStream(InputStream is)
Reads a frame block from an input stream, using a fused buffered reader
with special support for matrix blocks.
|
| Modifier and Type | Method and Description |
|---|---|
static MatrixBlock |
DataConverter.convertToMatrixBlock(FrameBlock frame)
Converts a frame block with arbitrary schema into a matrix block.
|
static String[][] |
DataConverter.convertToStringFrame(FrameBlock frame)
Converts a frame block with arbitrary schema into a two dimensional
string array.
|
static String |
DataConverter.toString(FrameBlock fb) |
static String |
DataConverter.toString(FrameBlock fb,
boolean sparse,
String separator,
String lineseparator,
int rowsToPrint,
int colsToPrint,
int decimal) |
static void |
LocalFileUtils.writeFrameBlockToLocal(String fname,
FrameBlock fb)
Writes a frame block to local file system.
|
| Modifier and Type | Method and Description |
|---|---|
static IndexRange |
UtilFunctions.getSelectedRangeForZeroOut(Pair<Long,FrameBlock> in,
int blen,
IndexRange indexRange,
long lSrcRowIndex,
long lDestRowIndex) |
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
DataAugmentation.dataCorruption(FrameBlock input,
double pTypo,
double pMiss,
double pDrop,
double pOut,
double pSwap)
This function returns a new frame block with error introduced in the data:
Typos in string values, null values, outliers in numeric data and swapped elements.
|
static FrameBlock |
DataAugmentation.miss(FrameBlock frame,
double pMiss,
double pDrop)
This function modifies the given, preprocessed frame block to add missing values to some of the rows,
marking them with the label missing.
|
static FrameBlock |
DataAugmentation.outlier(FrameBlock frame,
List<Integer> numerics,
double pOut,
double pPos,
int times)
This function modifies the given, preprocessed frame block to add outliers to some
of the numeric data of the frame, adding or several times the standard deviation,
and marking them with the label outlier.
|
static FrameBlock |
DataAugmentation.preprocessing(FrameBlock frame,
List<Integer> numerics,
List<Integer> strings,
List<Integer> swappable)
This function returns a new frame block with a labels column added, and build the lists
with column index of the different types of data.
|
static FrameBlock |
DataAugmentation.swap(FrameBlock frame,
List<Integer> swappable,
double pSwap)
This function modifies the given, preprocessed frame block to add swapped fields of the same ValueType
that are consecutive, marking them with the label swap.
|
static FrameBlock |
DataAugmentation.typos(FrameBlock frame,
List<Integer> strings,
double pTypo)
This function modifies the given, preprocessed frame block to add typos to the string values,
marking them with the label typos.
|
| Modifier and Type | Method and Description |
|---|---|
static FrameBlock |
DataAugmentation.dataCorruption(FrameBlock input,
double pTypo,
double pMiss,
double pDrop,
double pOut,
double pSwap)
This function returns a new frame block with error introduced in the data:
Typos in string values, null values, outliers in numeric data and swapped elements.
|
static FrameBlock |
DataAugmentation.miss(FrameBlock frame,
double pMiss,
double pDrop)
This function modifies the given, preprocessed frame block to add missing values to some of the rows,
marking them with the label missing.
|
static FrameBlock |
DataAugmentation.outlier(FrameBlock frame,
List<Integer> numerics,
double pOut,
double pPos,
int times)
This function modifies the given, preprocessed frame block to add outliers to some
of the numeric data of the frame, adding or several times the standard deviation,
and marking them with the label outlier.
|
static FrameBlock |
DataAugmentation.preprocessing(FrameBlock frame,
List<Integer> numerics,
List<Integer> strings,
List<Integer> swappable)
This function returns a new frame block with a labels column added, and build the lists
with column index of the different types of data.
|
static FrameBlock |
DataAugmentation.swap(FrameBlock frame,
List<Integer> swappable,
double pSwap)
This function modifies the given, preprocessed frame block to add swapped fields of the same ValueType
that are consecutive, marking them with the label swap.
|
static FrameBlock |
DataAugmentation.typos(FrameBlock frame,
List<Integer> strings,
double pTypo)
This function modifies the given, preprocessed frame block to add typos to the string values,
marking them with the label typos.
|
Copyright © 2020 The Apache Software Foundation. All rights reserved.