de.erichseifert.gral.util

Class StatefulTokenizer

java.lang.Object

de.erichseifert.gral.util.StatefulTokenizer

public abstract class StatefulTokenizer
extends java.lang.Object

A tokenizing parser that can analyzes a string using different sets of regular expression based parsing rules and produces a list of tokens. The class is intended to be sub-classed to implement new grammars. Different sets of rules can be defined with the method putRules(String, Rule[]), e.g. for string processing. Each rule in a set produces one token with a type name and it can switch to another state or switch back to the previous state with the special state "#pop". The is a list of tokens with arbitrary type. The list of produced tokens can be filtered: tokens of same type can be joined by adding the type with addJoinedType(Object) and tokens can be omitted from the result for easier post-processing by adding with addIgnoredType(Object).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StatefulTokenizer.Token A token that designates a certain section of a text input.

Method Summary

Modifier and Type	Method and Description
java.util.List<StatefulTokenizer.Token>	tokenize(java.lang.String data) Analyzes the specified input string using different sets of rules and returns a list of token objects describing the content structure.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

tokenize

public java.util.List<StatefulTokenizer.Token> tokenize(java.lang.String data)

Analyzes the specified input string using different sets of rules and returns a list of token objects describing the content structure.

Parameters:

data - Input string.

Returns:

List of tokens.