org.apache.sysds.runtime.transform.tokenize

Class TokenizerPreNgram

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.TokenizerPreNgram

All Implemented Interfaces:

Serializable, TokenizerPre

public class TokenizerPreNgram
extends Object
implements TokenizerPre

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
TokenizerPreNgram(List<Integer> idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)

Method Summary

Modifier and Type	Method and Description
List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens>	tokenizePre(FrameBlock in)
List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>	wordTokenListToNgrams(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens)
List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token>	wordTokenToNgrams(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerPreNgram

public TokenizerPreNgram(List<Integer> idCols,
                         int tokenizeCol,
                         org.apache.wink.json4j.JSONObject params)
                  throws org.apache.wink.json4j.JSONException

Throws:

org.apache.wink.json4j.JSONException

Method Detail

wordTokenToNgrams

public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenToNgrams(org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token wordTokens)

wordTokenListToNgrams

public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokenListToNgrams(List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.Token> wordTokens)

tokenizePre

public List<org.apache.sysds.runtime.transform.tokenize.Tokenizer.DocumentToTokens> tokenizePre(FrameBlock in)

Specified by:

tokenizePre in interface TokenizerPre