Tokenizer

Members

CHANNEL_PREFIX :string

Source:

• tokenizer.js, line 13

Type:

• string

USER_PREFIX :string

Source:

• tokenizer.js, line 17

Type:

• string

SEPARATOR_CHARACTERS :Array

Source:

• tokenizer.js, line 21

Type:

• Array

CHANNEL_ALLOWED_CHARACTERS :array

Source:

• tokenizer.js, line 25

Type:

• array

USER_ALLOWED_CHARACTERS :array

Source:

• tokenizer.js, line 29

Type:

• array

Methods

(static) tokenize(text) → {Array.<Token>}

Source:

• tokenizer.js, line 35

Parameters:

Returns:

Type

Array.<Token>