Natural Language - Token (Word|Term)

About

In Natural Language processing, Tokens can be things like:

• word,
• numbers,
• acronyms,
• word-roots
• or fixed-length character strings.

A token is the result of parsing (tokenization) the document down to the atomic elements generally of a language.

The token are then searchable.

See:

• English - Parts of the speech
• Text Mining - Term

Articles Related

Management

Create

See Natural Language Processing - (Tokenization|Parser|Text Segmentation|Word Break rules|Text Analysis)

Tokenization

See Natural Language Processing - (Tokenization|Parser|Text Segmentation|Word Break rules|Text Analysis)

Stemming

NLP - (Word Stem|Stemming)

Visualize

• https://github.com/wooorm/common-words - Rare word visualization