smile.nlp.tokenizer

Class BreakIteratorTokenizer

java.lang.Object

smile.nlp.tokenizer.BreakIteratorTokenizer

All Implemented Interfaces:

Tokenizer

public class BreakIteratorTokenizer
extends java.lang.Object
implements Tokenizer

A word tokenizer based on the java.text.BreakIterator, which supports multiple natural languages (selected by locale setting).

Constructor Summary

Constructors

Constructor and Description
BreakIteratorTokenizer() Constructor for the default locale.
BreakIteratorTokenizer(java.util.Locale locale) Constructor for the given locale.

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	split(java.lang.String text) Divide the given string into a list of substrings.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BreakIteratorTokenizer

public BreakIteratorTokenizer()

Constructor for the default locale.

BreakIteratorTokenizer

public BreakIteratorTokenizer(java.util.Locale locale)

Constructor for the given locale.

Method Detail

split

public java.lang.String[] split(java.lang.String text)

Description copied from interface: Tokenizer

Divide the given string into a list of substrings.

Specified by:

split in interface Tokenizer