cc.mallet.pipe

Class NGramPreprocessor

java.lang.Object

cc.mallet.pipe.Pipe

cc.mallet.pipe.NGramPreprocessor

All Implemented Interfaces:

AlphabetCarrying, java.io.Serializable

public class NGramPreprocessor
extends Pipe
implements java.io.Serializable

This pipe changes text to lowercase, removes common XML entities (quot, apos, lt, gt), and replaces all punctuation except the - character with whitespace. It then breaks up tokens on whitespace and applies n-gram token replacements and deletions. Replacements are applied in the order they are specified, first by file and then within files.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	NGramPreprocessor.ReplacementSet

Field Summary

Fields

Modifier and Type	Field and Description
java.util.ArrayList<NGramPreprocessor.ReplacementSet>	replacementSets

Constructor Summary

Constructors

Constructor and Description
NGramPreprocessor()

Method Summary

Modifier and Type	Method and Description
int	loadDeletions(java.lang.String filename)
int	loadReplacements(java.lang.String filename)
Instance	pipe(Instance instance) Really this should be 'protected', but isn't for historical reasons.

Methods inherited from class cc.mallet.pipe.Pipe

alphabetsMatch, getAlphabet, getAlphabets, getDataAlphabet, getInstanceId, getTargetAlphabet, instanceFrom, instancesFrom, instancesFrom, isDataAlphabetSet, isTargetProcessing, newIteratorFrom, precondition, readResolve, setDataAlphabet, setOrCheckDataAlphabet, setOrCheckTargetAlphabet, setTargetAlphabet, setTargetProcessing

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

replacementSets

public java.util.ArrayList<NGramPreprocessor.ReplacementSet> replacementSets

Constructor Detail

NGramPreprocessor

public NGramPreprocessor()

Method Detail

loadReplacements

public int loadReplacements(java.lang.String filename)
                     throws java.io.IOException

Throws:

java.io.IOException

loadDeletions

public int loadDeletions(java.lang.String filename)
                  throws java.io.IOException

Throws:

java.io.IOException

pipe

public Instance pipe(Instance instance)

Description copied from class: Pipe

Really this should be 'protected', but isn't for historical reasons.

Overrides:

pipe in class Pipe