smile.nlp.keyword

Class CooccurrenceKeywordExtractor

java.lang.Object

smile.nlp.keyword.CooccurrenceKeywordExtractor

public class CooccurrenceKeywordExtractor
extends java.lang.Object

Keyword extraction from a single document using word co-occurrence statistical information. The algorithm was proposed by Y. Matsuo and M. Ishizuka. It consists of six steps:

1. Stem words by Porter algorithm and extract phrases based APRIORI algorithm (upto 4 words with frequency more than 3 times). Discard stop words.
2. Select the top frequent terms up to 30% of running terms.
3. Clustering frequent terms. Two terms are in the same cluster if either their Jensen-Shannon divergence or mutual information is above the threshold (0.95 * log 2, and log 2, respectively).
4. Calculate the expected co-occurrence probability
5. Calculate the refined χ2 values that removes the maximal term.
6. Output a given number of terms of largest refined χ2 values.

Constructor Summary

Constructors

Constructor and Description
CooccurrenceKeywordExtractor()

Method Summary

Modifier and Type	Method and Description
java.util.ArrayList<NGram>	extract(java.lang.String text) Returns the top 10 keywords.
java.util.ArrayList<NGram>	extract(java.lang.String text, int maxNumKeywords) Returns a given number of top keywords.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CooccurrenceKeywordExtractor

public CooccurrenceKeywordExtractor()

Method Detail

extract

public java.util.ArrayList<NGram> extract(java.lang.String text)

Returns the top 10 keywords.

Parameters:

text - A single document.

Returns:

The top 10 keywords.

extract

public java.util.ArrayList<NGram> extract(java.lang.String text,
                                          int maxNumKeywords)

Returns a given number of top keywords.

Parameters:

text - A single document.

Returns:

The top keywords.