Package org.apache.sysds.runtime.transform.tokenize

Class TokenizerPostPosition

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.TokenizerPostPosition

All Implemented Interfaces:

Serializable, TokenizerPost

public class TokenizerPostPosition
extends Object
implements TokenizerPost

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
TokenizerPostPosition(org.apache.wink.json4j.JSONObject params, int numIdCols, int maxTokens, boolean wideFormat)

Method Summary

Modifier and Type	Method	Description
void	appendTokensLong(List<Object> keys, List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> tokenList, FrameBlock out)
void	appendTokensWide(List<Object> keys, List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> tokenList, FrameBlock out)
long	getNumCols()
long	getNumRows(long inRows)
Types.ValueType[]	getOutSchema()
FrameBlock	tokenizePost(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tl, FrameBlock out)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerPostPosition

public TokenizerPostPosition(org.apache.wink.json4j.JSONObject params,
                             int numIdCols,
                             int maxTokens,
                             boolean wideFormat)

Method Detail

tokenizePost

public FrameBlock tokenizePost(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tl,
                               FrameBlock out)

Specified by:

tokenizePost in interface TokenizerPost

appendTokensLong

public void appendTokensLong(List<Object> keys,
                             List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> tokenList,
                             FrameBlock out)

appendTokensWide

public void appendTokensWide(List<Object> keys,
                             List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> tokenList,
                             FrameBlock out)

getOutSchema

public Types.ValueType[] getOutSchema()

Specified by:

getOutSchema in interface TokenizerPost

getNumRows

public long getNumRows(long inRows)

Specified by:

getNumRows in interface TokenizerPost

getNumCols

public long getNumCols()

Specified by:

getNumCols in interface TokenizerPost