GermanStemmer

Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: A stemmer for German texts.

Description:

The WVTool Tutorial

42CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE 8.1.6 GermanStopwordFilter

Group: IO.Text.Filter Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Standard stopwords list for German texts.

Description:

8.1.7 LogFileSource Group: IO.Web

Generated output:

ˆ ExampleSet Parameters:

ˆ cong_le: the format conguration le (lename)

ˆ log_dir: the directory containing the log les (lename)

ž dns_lookup: Perform reverse dns lookup on the client ip (boolean; default:

false)

ž robot_lter: le that contains regular expressions on user agents that should be ltered out. Each line must contain exactly one regular ex-pression. (lename)

ž letype_lter: le that contains regular expressions on les that should be ltered out. Each line must contain exactly one regular expression.

(string)

ž only_HTTP_200: Consider only entries with HTTP Response code 200 (boolean; default: false)

8.1. TEXT 43

ž browser_matcher: le that contains regular expressions to match browser types. Each line must contain exactly an expression of the form <name>:<regular expression>. (list)

ž os_matcher: le that contains regular expressions to match os types. Each line must contain exactly an expression of the form <name>:<regular expression>. (list)

ž language_matcher: le that contains regular expressions to match

lan-guages. Each line must contain exactly an expression of the form <name>:<regular expression>. (list)

ž session_timeout: Time between two requests from the same source, such that the second request can be assumed to be a new session (integer;

0-+∞; default: 400000) Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Reads a web server log le.

Description:

8.1.8 LovinsStemmer Group: IO.Text.Stemmer Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: The Lovins stemmer for English texts.

The WVTool Tutorial

44CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR

ž attributes: Species a list of attribute names and extraction queries. These queries can be XPath or a regular expression. If a regular expression is used, the query must have the following form: '<regex-expression>

<replacement-pattern>', where the <replacement_pattern> states how a match is replaced to generate the nal information. '$1' would yield the rst matching group as result. A number sign in front of an attribute name marks the attribute as numeric. In these cases, the operator uses dierent heuristicts to parse a number from the extracted string. An ! in front of an attribute name marks it as binary. For both XPath and regex, only the rst match is used. (list)

ž namespaces: Species pairs of identier and namespace for use in XPath queries. The namespace for (x)html is bound automatically to the identier h. (list)

ž url: The url of the HTTP GET based service. This URL may contain terms of the form <attributeName> that are replaced by the value of the corresonding attribute before invoking the query. (string)

ž separators: Characters used to separate entries in the result eld obtained by XPath or regular expression. (string)

ž delay: Amount of milliseconds to wait between requests (integer; 0-+∞; default: 0)

Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Extracts information from a HTTP GET based web re-source.

8.1. TEXT 45 Description:

8.1.10 NGramTokenizer

Group: IO.Text.Tokenizer Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Parameters:

ž length: The maximal length of the ngrams. (integer; 1-+∞; default: 3)

ž keep_terms: Indicates if the original terms should be kept along with the ngrams. (boolean; default: false)

Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Creates ngrams of the input token stream.

Description:

8.1.11 PorterStemmer

Group: IO.Text.Stemmer Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

The WVTool Tutorial

46CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE Short description: The Porter stemmer for English texts.

Description:

8.1.12 Segmenter Group: IO.Text.Misc

Parameters:

ž preview: Shows a preview for the results which will be achieved by the current conguration.

ˆ texts: A directory containing the documents to be segmented (lename)

ž content_type: The content type of the input texts (txt, xml, html) (string)

ˆ output: The directory to which to write the segments (lename)

ˆ expression: Species a regular expression or XPath expression that matches against substrings of the content which should be treated as individual segments. The syntax is the same as for attribute extraction (see WVTool operator), but instead of extracting only the rst match, all matches are extracted and written to individual les (string)

ž ignore_cdata: Species whether CDATA should be ignored when parsing HTML (boolean; default: true)

ž namespaces: Species pairs of identier and namespace for use in XPath queries. The namespace for (x)html is bound automatically to the identier h. (list)

Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Segments documents based on regular expressions or xpath.

Description:

8.1. TEXT 47 8.1.13 ServerLog2Transactions

Group: IO.Web Required input:

ˆ ExampleSet

Generated output:

ˆ ExampleSet Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Converts an example set containing a server log to trans-actions

Description:

8.1.14 SingleTextInput

Group: IO.Text Generated output:

ˆ ExampleSet

ˆ WordList Parameters:

ˆ text: The input text. (string)

ž default_content_type: The default content type if not specied by the example set (possible values: pdf, html, htm, xml, text, txt). (string;

default: )

ž default_content_encoding: The default content encoding if not speci-ed by the example set (only encodings supportspeci-ed by Java can be usspeci-ed).

(string; default: )

ž default_content_language: The default content language if not speci-ed by the example set. (string; default: )

The WVTool Tutorial

48CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE

ž prune_below: Prune words that appear inat most that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the lowest document frequency in p words with the highest frequency.

(string; default: '-1')

ž prune_above: Prune words that appear in at least that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the highest document frequency in p words with the lowest frequency.

(string; default: '-1')

ž vector_creation: Method used to create word vectors

ž use_content_attributes: If set to true, the returned example set will con-tain content type, encoding, and language attributes. (boolean; default:

false)

ž use_given_word_list: If set, the given word of list in the input will be used (boolean; default: false)

ž input_word_list: Load a word list from this le instead of creating it from the input data. (lename)

ž return_word_list: If checked the word list will be returned as part of the result. (boolean; default: false)

ž output_word_list: Save the used word list into this le. (lename)

ˆ id_attribute_type: Indicates if long ids (complete paths), short ids (last part of the source name), or numerical ids will be used.

ž namespaces: Species pairs of identier and namespace for use in XPath queries. The namespace for (x)html is bound automatically to the identier h. (list)

ž text_query: Query that extracts the parts of a document, that should be used for vectorization. This query can be XPath or a regular expres-sion. If a regular expression is used, the query must have the follow-ing form: '<regex-expression> <replacement-pattern>', where the <re-placement_pattern> states how a match is replaced to generate the nal information. '$1' would yield the rst matching group as result. For both, XPath and regular expression, all matches are concatanated and then passed to the vectorization process. (string)

ž create_text_visualizer: Indicates if a text specic object visualizer should be created which can be used in plotters etc. Note: Text visualization does not work for id type number. (boolean; default: false)

ž on_the_y_pruning: Denotes after how many documents, singular terms should be removed from the word list. 0 indicates no pruning. (integer;

0-+∞; default: -1) Values:

8.1. TEXT 49

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Inner operators: The inner operators must be able to handle [TokenSequence]

and must deliver [TokenSequence].

Short description: Generates word vectors from a single text.

Description:

8.1.15 SnowballStemmer

Group: IO.Text.Stemmer Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: The Snowball stemmer for dierent languages.

Description:

8.1.16 SplitSegmenter

Group: IO.Text.Misc Parameters:

ž preview: Shows a preview for the results which will be achieved by the current conguration.

The WVTool Tutorial

50CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE

ˆ texts: A directory containing the documents to be segmented (lename)

ˆ output: The directory to which to write the segments (lename)

ˆ split_expression: Species a regular expression or XPath expression that matches against substrings of the content which should be treated as individual segments. The syntax is the same as for attribute extraction (see WVTool operator), but instead of extracting only the rst match, all matches are extracted and written to individual les (string)

Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Segments documents by dening the splitting point.

Description:

8.1.17 StopwordFilterFile Group: IO.Text.Filter

Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Parameters:

ˆ le: File that contains the stopwords one per line (lename)

ž case_sensitive: Should words be matched case sensitive (boolean; default:

false) Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Filters terms based on a list of expressions provided in an external le.

8.1. TEXT 51

ž lter_nominal_attributes: Indicates if nominal attributes should also be ltered in addition to string attributes. (boolean; default: false)

ž remove_original_attributes: Indicates if the original nominal and / or string attributes should also be removed after the word vector creation.

(boolean; default: false)

ž default_content_type: The default content type if not specied by the example set (possible values: pdf, html, htm, xml, text, txt). (string;

default: )

ž default_content_encoding: The default content encoding if not speci-ed by the example set (only encodings supportspeci-ed by Java can be usspeci-ed).

(string; default: )

ž default_content_language: The default content language if not speci-ed by the example set. (string; default: )

ž prune_below: Prune words that appear inat most that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the lowest document frequency in p words with the highest frequency.

(string; default: '-1')

ž prune_above: Prune words that appear in at least that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the highest document frequency in p words with the lowest frequency.

(string; default: '-1')

ž vector_creation: Method used to create word vectors

ž use_content_attributes: If set to true, the returned example set will con-tain content type, encoding, and language attributes. (boolean; default:

false)

ž use_given_word_list: If set, the given word of list in the input will be used (boolean; default: false)

The WVTool Tutorial

52CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE

ž input_word_list: Load a word list from this le instead of creating it from the input data. (lename)

ž return_word_list: If checked the word list will be returned as part of the result. (boolean; default: false)

ž output_word_list: Save the used word list into this le. (lename)

ˆ id_attribute_type: Indicates if long ids (complete paths), short ids (last part of the source name), or numerical ids will be used.

ž namespaces: Species pairs of identier and namespace for use in XPath queries. The namespace for (x)html is bound automatically to the identier h. (list)

ž text_query: Query that extracts the parts of a document, that should be used for vectorization. This query can be XPath or a regular expres-sion. If a regular expression is used, the query must have the follow-ing form: '<regex-expression> <replacement-pattern>', where the <re-placement_pattern> states how a match is replaced to generate the nal information. '$1' would yield the rst matching group as result. For both, XPath and regular expression, all matches are concatanated and then passed to the vectorization process. (string)

ž create_text_visualizer: Indicates if a text specic object visualizer should be created which can be used in plotters etc. Note: Text visualization does not work for id type number. (boolean; default: false)

ž on_the_y_pruning: Denotes after how many documents, singular terms should be removed from the word list. 0 indicates no pruning. (integer;

0-+∞; default: -1) Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Inner operators: The inner operators must be able to handle [TokenSequence]

and must deliver [TokenSequence].

Short description: Generates word vectors from string attributes.

Description:

8.1. TEXT 53 8.1.19 StringTokenizer

Group: IO.Text.Tokenizer Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Tokenizes a set of input tokens.

Description:

8.1.20 TagLogSource

Group: IO.Web Generated output:

ˆ ExampleSet Parameters:

ˆ tag_logle: the tag log le (lename)

ž min_occurrences: minimal number of occurrences of a tag to be consid-ered (integer; 1-+∞; default: 100)

Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Reads a tag log le.

The WVTool Tutorial

54CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE Description:

8.1.21 TermNGramGenerator Group: IO.Text.Tokenizer

Required input:

ˆ TokenSequence

Generated output:

ˆ TokenSequence Parameters:

ž max_length: The maximal length of the ngrams. (integer; 1-+∞; default:

2) Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Short description: Creates term ngrams of the input token stream.

Description:

8.1.22 TextInput

Group: IO.Text Generated output:

ˆ ExampleSet

ˆ WordList Parameters:

ž texts: Species a list of class/directory pairs. (list)

ž default_content_type: The default content type if not specied by the example set (possible values: pdf, html, htm, xml, text, txt). (string;

default: )

8.1. TEXT 55

ž default_content_encoding: The default content encoding if not speci-ed by the example set (only encodings supportspeci-ed by Java can be usspeci-ed).

(string; default: )

ž default_content_language: The default content language if not speci-ed by the example set. (string; default: )

ž prune_below: Prune words that appear inat most that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the lowest document frequency in p words with the highest frequency.

(string; default: '-1')

ž prune_above: Prune words that appear in at least that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the highest document frequency in p words with the lowest frequency.

(string; default: '-1')

ž vector_creation: Method used to create word vectors

ž use_content_attributes: If set to true, the returned example set will con-tain content type, encoding, and language attributes. (boolean; default:

false)

ž use_given_word_list: If set, the given word of list in the input will be used (boolean; default: false)

ž input_word_list: Load a word list from this le instead of creating it from the input data. (lename)

ž return_word_list: If checked the word list will be returned as part of the result. (boolean; default: false)

ž output_word_list: Save the used word list into this le. (lename)

ˆ id_attribute_type: Indicates if long ids (complete paths), short ids (last part of the source name), or numerical ids will be used.

ž namespaces: Species pairs of identier and namespace for use in XPath queries. The namespace for (x)html is bound automatically to the identier h. (list)

ž text_query: Query that extracts the parts of a document, that should be used for vectorization. This query can be XPath or a regular expres-sion. If a regular expression is used, the query must have the follow-ing form: '<regex-expression> <replacement-pattern>', where the <re-placement_pattern> states how a match is replaced to generate the nal information. '$1' would yield the rst matching group as result. For both, XPath and regular expression, all matches are concatanated and then passed to the vectorization process. (string)

ž create_text_visualizer: Indicates if a text specic object visualizer should be created which can be used in plotters etc. Note: Text visualization does not work for id type number. (boolean; default: false)

The WVTool Tutorial

56CHAPTER 8. APPENDIX B - RAPIDMINER TEXT PLUGIN OPERATOR REFERENCE

ž on_the_y_pruning: Denotes after how many documents, singular terms should be removed from the word list. 0 indicates no pruning. (integer;

0-+∞; default: -1)

ž extend_exampleset: If true, an input example set is not only used to specify the documents that should be vectorized, but this example set is merged with the vectors. Note, that this works only with nominal ids!

(boolean; default: false) Values:

ˆ applycount: The number of times the operator was applied.

ˆ looptime: The time elapsed since the current loop started.

ˆ time: The time elapsed since this operator started.

Inner operators: The inner operators must be able to handle [TokenSequence]

and must deliver [TokenSequence].

Short description: Generates word vectors from text collections.

Description:

8.1.23 TextObjectTextInput Group: IO.Text

Generated output:

ˆ ExampleSet

ˆ WordList Parameters:

ž default_content_type: The default content type if not specied by the example set (possible values: pdf, html, htm, xml, text, txt). (string;

default: )

ž default_content_encoding: The default content encoding if not speci-ed by the example set (only encodings supportspeci-ed by Java can be usspeci-ed).

(string; default: )

ž default_content_language: The default content language if not speci-ed by the example set. (string; default: )

8.1. TEXT 57

ž prune_below: Prune words that appear inat most that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the lowest document frequency in p words with the highest frequency.

(string; default: '-1')

ž prune_above: Prune words that appear in at least that many documents.

-1 for no pruning. Alternatively you can provide a percentage value, denot-ing the highest document frequency in p words with the lowest frequency.

(string; default: '-1')

ž vector_creation: Method used to create word vectors

ž use_content_attributes: If set to true, the returned example set will con-tain content type, encoding, and language attributes. (boolean; default:

false)

ž use_given_word_list: If set, the given word of list in the input will be used (boolean; default: false)

ž input_word_list: Load a word list from this le instead of creating it from

ž input_word_list: Load a word list from this le instead of creating it from

Im Dokument The Word Vector Tool and the RapidMiner Text Plugin (Seite 41-0)