Tokenizers

The tokenizers module gathers the library’s various tokenizers.

Tokenizers are algorithms whose goal is to take some raw text & split this one into tokens.

Splitting a text into a list of sentences, or sentences into a list of words are both examples of what a tokenizer is.

Summary

Modules under the talisman/tokenizers namespace:

• hyphenation
• lines
• ngrams
• paragraphs
• sentences
• skipgrams
• syllables
• tweets
• words