org.apache.sysds.runtime.transform.tokenize

Class TokenizerPreWhitespaceSplit

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.TokenizerPreWhitespaceSplit

All Implemented Interfaces:

Serializable, TokenizerPre

public class TokenizerPreWhitespaceSplit
extends Object
implements TokenizerPre

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
TokenizerPreWhitespaceSplit(List<Integer> idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)

Method Summary

Modifier and Type	Method and Description
List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>	splitToTokens(String text)
List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>	tokenizePre(FrameBlock in)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerPreWhitespaceSplit

public TokenizerPreWhitespaceSplit(List<Integer> idCols,
                                   int tokenizeCol,
                                   org.apache.wink.json4j.JSONObject params)
                            throws org.apache.wink.json4j.JSONException

Throws:

org.apache.wink.json4j.JSONException

Method Detail

splitToTokens

public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> splitToTokens(String text)

tokenizePre

public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre(FrameBlock in)

Specified by:

tokenizePre in interface TokenizerPre